[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:31:29 dist-test-slave-6224 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:31:29 dist-test-slave-6224 krb5kdc[399](info): set up 2 sockets
Jun 18 23:31:29 dist-test-slave-6224 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260618 23:31:30.667999 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:31:30 dist-test-slave-6224 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:31:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:31:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:31:30.800299 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45503
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45503
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:30.891463 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:30.891757 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:30.891783 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:30.893100 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:31:30.893149 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:30.893164 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:30.893177 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:31:30.893191 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:31:30.894795 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:30.895084 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:30.895399 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:30.901496 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:30.901566 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:30.901468 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:30.901454 412 server_base.cc:1034] running on GCE node
I20260618 23:31:30.902010 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:30.902383 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:30.903644 412 hybrid_clock.cc:648] HybridClock initialized: now 1781825490903593 us; error 64 us; skew 500 ppm
Jun 18 23:31:30 dist-test-slave-6224 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825490, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:30.909289 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:31:30.909808 412 webserver.cc:460] Webserver started at http://127.0.98.62:43241/ using document root <none> and password file <none>
I20260618 23:31:30.910058 412 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:30.910108 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:30.910216 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:30.911356 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "5c507b6c467d48ad8d3eb2aac0720bb8"
format_stamp: "Formatted at 2026-06-18 23:31:30 on dist-test-slave-6224"
server_key: "0ad9eda1e99c3839e343ff28b1e4999e"
server_key_iv: "20d4fd1120f52907a40831f387895fc0"
server_key_version: "encryptionkey@0"
I20260618 23:31:30.911788 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "5c507b6c467d48ad8d3eb2aac0720bb8"
format_stamp: "Formatted at 2026-06-18 23:31:30 on dist-test-slave-6224"
server_key: "0ad9eda1e99c3839e343ff28b1e4999e"
server_key_iv: "20d4fd1120f52907a40831f387895fc0"
server_key_version: "encryptionkey@0"
I20260618 23:31:30.913368 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:31:30.914254 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:30.914549 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:31:30.914675 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "5c507b6c467d48ad8d3eb2aac0720bb8"
format_stamp: "Formatted at 2026-06-18 23:31:30 on dist-test-slave-6224"
server_key: "0ad9eda1e99c3839e343ff28b1e4999e"
server_key_iv: "20d4fd1120f52907a40831f387895fc0"
server_key_version: "encryptionkey@0"
I20260618 23:31:30.914759 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:30.942261 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:30.944741 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:30.944945 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:30.949550 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45503
I20260618 23:31:30.949615 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45503 every 8 connection(s)
I20260618 23:31:30.950017 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:31:30.951130 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:30.953958 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8: Bootstrap starting.
I20260618 23:31:30.954655 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:30.954995 480 log.cc:824] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:30.956310 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8: No bootstrap required, opened a new log
I20260618 23:31:30.956570 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 412
I20260618 23:31:30.956674 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:31:30.957490 392 external_mini_cluster.cc:1428] Setting key 20f3c78bc3b61213c969d5029bceb3b4
I20260618 23:31:30.958971 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c507b6c467d48ad8d3eb2aac0720bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } }
I20260618 23:31:30.959157 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:30.959209 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c507b6c467d48ad8d3eb2aac0720bb8, State: Initialized, Role: FOLLOWER
I20260618 23:31:30.959352 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c507b6c467d48ad8d3eb2aac0720bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } }
I20260618 23:31:30.959434 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:31:30.959460 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:31:30.959533 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:30.960512 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c507b6c467d48ad8d3eb2aac0720bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } }
I20260618 23:31:30.960610 480 leader_election.cc:302] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c507b6c467d48ad8d3eb2aac0720bb8; no voters:
I20260618 23:31:30.960870 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:30.960855 480 leader_election.cc:288] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:31:30.961042 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8 [term 1 LEADER]: Becoming Leader. State: Replica: 5c507b6c467d48ad8d3eb2aac0720bb8, State: Running, Role: LEADER
I20260618 23:31:30.961130 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c507b6c467d48ad8d3eb2aac0720bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } }
I20260618 23:31:30.961253 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:31:30.961516 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c507b6c467d48ad8d3eb2aac0720bb8. Latest consensus state: current_term: 1 leader_uuid: "5c507b6c467d48ad8d3eb2aac0720bb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c507b6c467d48ad8d3eb2aac0720bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } } }
I20260618 23:31:30.961745 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8 [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:30.961684 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c507b6c467d48ad8d3eb2aac0720bb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c507b6c467d48ad8d3eb2aac0720bb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45503 } } }
I20260618 23:31:30.962081 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8 [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:30.962098 493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:31:30.962864 493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:31:30.964967 502 mvcc.cc:205] Tried to move back new op lower bound from 7298357210988019712 to 7298357210977107968. Current Snapshot: MvccSnapshot[applied={T|T < 7298357210988019712}]
I20260618 23:31:30.965448 493 catalog_manager.cc:1347] Generated new cluster ID: df8b5ace0b5b4f3aa211d36fb35750c4
I20260618 23:31:30.965512 493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 18 23:31:30 dist-test-slave-6224 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825490, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:30.974720 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:30.958668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56670 (local address 127.0.98.62:45503)
0618 23:31:30.959074 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:30.959083 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:30.959151 (+ 68us) server_negotiation.cc:407] Connection header received
0618 23:31:30.961452 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:30.961461 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:30.961525 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:30.961646 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:30.964119 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:30.965387 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:30.966755 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:30.967215 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:30.967345 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:30.970894 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:30.970899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:30.970899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:30.970918 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:30.972735 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:30.973193 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:30.973196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:30.973197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:30.973308 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:30.973579 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:30.973580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:30.973581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:30.973855 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:30.974040 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:30.974306 (+ 266us) server_negotiation.cc:299] Negotiation successful
0618 23:31:30.974453 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":141,"threads_started":1}
I20260618 23:31:30.975720 493 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:31:30.976433 493 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:31:30.979628 493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c507b6c467d48ad8d3eb2aac0720bb8: Generated new TSK 0
I20260618 23:31:30.979826 493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260618 23:31:31.011368 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45503
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:31.116894 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:31.117195 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:31.117251 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:31.118809 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:31.118892 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:31.118949 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:31:31.120826 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45503
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:31.121131 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:31.121407 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:31.127220 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:31.127210 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:31.127266 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:31.127593 506 server_base.cc:1034] running on GCE node
I20260618 23:31:31.127925 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:31.128190 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:31.129364 506 hybrid_clock.cc:648] HybridClock initialized: now 1781825491129345 us; error 38 us; skew 500 ppm
Jun 18 23:31:31 dist-test-slave-6224 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:31.132457 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:31:31.133028 506 webserver.cc:460] Webserver started at http://127.0.98.1:43107/ using document root <none> and password file <none>
I20260618 23:31:31.133296 506 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:31.133369 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:31.133538 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:31.134783 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "0a3cd1c3795a44a2b2a56e23ce3f1a17"
format_stamp: "Formatted at 2026-06-18 23:31:31 on dist-test-slave-6224"
server_key: "41b2db0fda4b1e8120bea64a49d8c623"
server_key_iv: "8c9b2e72c46dfe2882ceb815e4064a2d"
server_key_version: "encryptionkey@0"
I20260618 23:31:31.135219 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0a3cd1c3795a44a2b2a56e23ce3f1a17"
format_stamp: "Formatted at 2026-06-18 23:31:31 on dist-test-slave-6224"
server_key: "41b2db0fda4b1e8120bea64a49d8c623"
server_key_iv: "8c9b2e72c46dfe2882ceb815e4064a2d"
server_key_version: "encryptionkey@0"
I20260618 23:31:31.136930 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260618 23:31:31.138109 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:31.138372 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:31.138507 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "0a3cd1c3795a44a2b2a56e23ce3f1a17"
format_stamp: "Formatted at 2026-06-18 23:31:31 on dist-test-slave-6224"
server_key: "41b2db0fda4b1e8120bea64a49d8c623"
server_key_iv: "8c9b2e72c46dfe2882ceb815e4064a2d"
server_key_version: "encryptionkey@0"
I20260618 23:31:31.138598 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:31.151788 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:31.154115 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:31.154323 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:31.154613 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:31.155010 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:31.155071 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:31.155114 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:31.155138 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:31.162977 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37517
I20260618 23:31:31.163075 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37517 every 8 connection(s)
I20260618 23:31:31.163470 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:31:31.168032 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 506
I20260618 23:31:31.168146 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:31:31.168377 392 external_mini_cluster.cc:1428] Setting key 6b98f125f06134ab0a948c6063f2ec09
Jun 18 23:31:31 dist-test-slave-6224 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:31.174496 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:31.164464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53069 (local address 127.0.98.62:45503)
0618 23:31:31.164539 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:31.164541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:31.164764 (+ 223us) server_negotiation.cc:407] Connection header received
0618 23:31:31.165317 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:31.165319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:31.165336 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:31.165422 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:31.166532 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:31.167087 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:31.168306 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:31.168614 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:31.168683 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:31.171281 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:31.171283 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:31.171283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:31.171290 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:31.172857 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:31.173420 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:31.173421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:31.173422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:31.173476 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:31.173789 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:31.173790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:31.173790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:31.173897 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:31.173952 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:31.174332 (+ 380us) server_negotiation.cc:299] Negotiation successful
0618 23:31:31.174420 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 23:31:31.174829 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:31.164387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45503 (local address 127.0.98.1:53069)
0618 23:31:31.164611 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:31.164660 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:31.165191 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:31.165516 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:31.165520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:31.166161 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:31.166411 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:31.166415 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:31.167184 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:31.167185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:31.168125 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:31.168127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:31.168713 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:31.168713 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:31.168901 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:31.169520 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:31.169529 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:31.171177 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:31.172969 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:31.172976 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:31.172978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:31.173320 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:31.173556 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:31.173557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:31.173558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:31.173705 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:31.173972 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:31.173978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:31.174172 (+ 194us) client_negotiation.cc:769] Sending connection context
0618 23:31:31.174315 (+ 143us) client_negotiation.cc:240] Negotiation successful
0618 23:31:31.174522 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":120,"threads_started":1}
I20260618 23:31:31.175462 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45503
I20260618 23:31:31.175608 635 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:31.175845 635 heartbeater.cc:511] Master 127.0.98.62:45503 requested a full tablet report, sending...
I20260618 23:31:31.176535 444 ts_manager.cc:194] Registered new tserver with Master: 0a3cd1c3795a44a2b2a56e23ce3f1a17 (127.0.98.1:37517)
I20260618 23:31:31.177223 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53069
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260618 23:31:31.198354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45503
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:31.286384 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:31.286636 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:31.286677 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:31.287945 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:31.288018 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:31.288065 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:31:31.289564 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45503
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:31.289839 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:31.290076 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:31.295703 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:31.295733 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:31.295733 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:31.296173 640 server_base.cc:1034] running on GCE node
I20260618 23:31:31.296414 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:31.296665 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:31.297821 640 hybrid_clock.cc:648] HybridClock initialized: now 1781825491297812 us; error 33 us; skew 500 ppm
Jun 18 23:31:31 dist-test-slave-6224 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:31.300987 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:31:31.301553 640 webserver.cc:460] Webserver started at http://127.0.98.2:38255/ using document root <none> and password file <none>
I20260618 23:31:31.301839 640 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:31.301914 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:31.302073 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:31.303272 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "d4b99ba41b794ec69f0a9a23182a910d"
format_stamp: "Formatted at 2026-06-18 23:31:31 on dist-test-slave-6224"
server_key: "9d45748334624d96421d4712a0de0d26"
server_key_iv: "f3ec8ea8b112e5c42de6339e437c5fd3"
server_key_version: "encryptionkey@0"
I20260618 23:31:31.303720 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d4b99ba41b794ec69f0a9a23182a910d"
format_stamp: "Formatted at 2026-06-18 23:31:31 on dist-test-slave-6224"
server_key: "9d45748334624d96421d4712a0de0d26"
server_key_iv: "f3ec8ea8b112e5c42de6339e437c5fd3"
server_key_version: "encryptionkey@0"
I20260618 23:31:31.305322 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:31:31.306234 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:31.306509 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:31.306610 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "d4b99ba41b794ec69f0a9a23182a910d"
format_stamp: "Formatted at 2026-06-18 23:31:31 on dist-test-slave-6224"
server_key: "9d45748334624d96421d4712a0de0d26"
server_key_iv: "f3ec8ea8b112e5c42de6339e437c5fd3"
server_key_version: "encryptionkey@0"
I20260618 23:31:31.306707 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:31.314157 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:31.316586 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:31.316782 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:31.317056 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:31.317435 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:31.317493 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:31.317529 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:31.317552 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:31.326992 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33367
I20260618 23:31:31.327097 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33367 every 8 connection(s)
I20260618 23:31:31.327481 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
Jun 18 23:31:31 dist-test-slave-6224 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:31.334558 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 640
I20260618 23:31:31.334674 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:31:31.334867 392 external_mini_cluster.cc:1428] Setting key b76f5ea91e4867bc68376d388af4270c
I20260618 23:31:31.338711 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:31.328390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52793 (local address 127.0.98.62:45503)
0618 23:31:31.328464 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:31.328469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:31.328723 (+ 254us) server_negotiation.cc:407] Connection header received
0618 23:31:31.329334 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:31.329335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:31.329348 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:31.329409 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:31.330445 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:31.330967 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:31.332077 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:31.332277 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:31.332364 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:31.334974 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:31.334977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:31.334978 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:31.334992 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:31.336858 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:31.337431 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:31.337432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:31.337433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:31.337524 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:31.337806 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:31.337807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:31.337807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:31.338004 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:31.338098 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:31.338464 (+ 366us) server_negotiation.cc:299] Negotiation successful
0618 23:31:31.338596 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 23:31:31.339278 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:31.328370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45503 (local address 127.0.98.2:52793)
0618 23:31:31.328573 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:31.328617 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:31.329205 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:31.329494 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:31.329497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:31.330127 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:31.330331 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:31.330336 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:31.331061 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:31.331062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:31.331987 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:31.331989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:31.332374 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:31.332375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:31.332502 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:31.333111 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:31.333120 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:31.334816 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:31.336962 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:31.336970 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:31.336972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:31.337323 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:31.337615 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:31.337615 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:31.337616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:31.337727 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:31.338119 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:31.338125 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:31.338291 (+ 166us) client_negotiation.cc:769] Sending connection context
0618 23:31:31.338406 (+ 115us) client_negotiation.cc:240] Negotiation successful
0618 23:31:31.338551 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":81,"threads_started":1}
I20260618 23:31:31.340024 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45503
I20260618 23:31:31.340143 769 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:31.340363 769 heartbeater.cc:511] Master 127.0.98.62:45503 requested a full tablet report, sending...
I20260618 23:31:31.340932 444 ts_manager.cc:194] Registered new tserver with Master: d4b99ba41b794ec69f0a9a23182a910d (127.0.98.2:33367)
I20260618 23:31:31.341413 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52793
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260618 23:31:31.363423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45503
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:31.446964 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:31.447186 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:31.447211 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:31.448365 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:31.448432 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:31.448482 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:31:31.449956 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45503
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:31.450152 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:31.450335 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:31.456092 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:31.456141 774 server_base.cc:1034] running on GCE node
W20260618 23:31:31.456280 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:31.456147 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:31.456621 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:31.456982 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:31.458168 774 hybrid_clock.cc:648] HybridClock initialized: now 1781825491458152 us; error 45 us; skew 500 ppm
Jun 18 23:31:31 dist-test-slave-6224 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:31.461272 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:31:31.461889 774 webserver.cc:460] Webserver started at http://127.0.98.3:35779/ using document root <none> and password file <none>
I20260618 23:31:31.462225 774 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:31.462280 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:31.462388 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:31.463734 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "807b783ee66f448c817728b7f14516cd"
format_stamp: "Formatted at 2026-06-18 23:31:31 on dist-test-slave-6224"
server_key: "4771bf1f8304d72b652456c4a2f2b85a"
server_key_iv: "a746a0e8f9631a21593d8a347bafdeb0"
server_key_version: "encryptionkey@0"
I20260618 23:31:31.464100 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "807b783ee66f448c817728b7f14516cd"
format_stamp: "Formatted at 2026-06-18 23:31:31 on dist-test-slave-6224"
server_key: "4771bf1f8304d72b652456c4a2f2b85a"
server_key_iv: "a746a0e8f9631a21593d8a347bafdeb0"
server_key_version: "encryptionkey@0"
I20260618 23:31:31.465709 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260618 23:31:31.466566 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:31.466895 774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:31.467063 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "807b783ee66f448c817728b7f14516cd"
format_stamp: "Formatted at 2026-06-18 23:31:31 on dist-test-slave-6224"
server_key: "4771bf1f8304d72b652456c4a2f2b85a"
server_key_iv: "a746a0e8f9631a21593d8a347bafdeb0"
server_key_version: "encryptionkey@0"
I20260618 23:31:31.467175 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:31.484835 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:31.487494 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:31.487726 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:31.488027 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:31.488389 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:31.488447 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:31.488497 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:31.488521 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:31.498356 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43239
I20260618 23:31:31.498495 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43239 every 8 connection(s)
I20260618 23:31:31.498857 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:31:31.499372 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 774
I20260618 23:31:31.499470 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:31:31.499681 392 external_mini_cluster.cc:1428] Setting key 6d5b9535a92efd014f0e7cee88d89270
Jun 18 23:31:31 dist-test-slave-6224 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825491, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:31.510186 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:31.500210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35081 (local address 127.0.98.62:45503)
0618 23:31:31.500260 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:31.500262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:31.500274 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:31:31.500830 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:31.500831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:31.500844 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:31.500906 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:31.501981 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:31.502526 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:31.503692 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:31.503913 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:31.504000 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:31.506429 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:31.506431 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:31.506431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:31.506439 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:31.508172 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:31.508910 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:31.508912 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:31.508912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:31.508994 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:31.509311 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:31.509312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:31.509313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:31.509461 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:31.509541 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:31.509958 (+ 417us) server_negotiation.cc:299] Negotiation successful
0618 23:31:31.510085 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260618 23:31:31.510250 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:31.499883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45503 (local address 127.0.98.3:35081)
0618 23:31:31.500056 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:31.500107 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:31.500710 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:31.500998 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:31.501002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:31.501643 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:31.501877 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:31.501882 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:31.502638 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:31.502640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:31.503510 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:31.503513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:31.504008 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:31.504009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:31.504144 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:31.504753 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:31.504763 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:31.506313 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:31.508412 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:31.508421 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:31.508423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:31.508784 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:31.509099 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:31.509101 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:31.509101 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:31.509235 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:31.509528 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:31.509534 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:31.509705 (+ 171us) client_negotiation.cc:769] Sending connection context
0618 23:31:31.509854 (+ 149us) client_negotiation.cc:240] Negotiation successful
0618 23:31:31.510029 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260618 23:31:31.510901 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45503
I20260618 23:31:31.511072 903 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:31.511374 903 heartbeater.cc:511] Master 127.0.98.62:45503 requested a full tablet report, sending...
I20260618 23:31:31.511940 443 ts_manager.cc:194] Registered new tserver with Master: 807b783ee66f448c817728b7f14516cd (127.0.98.3:43239)
I20260618 23:31:31.512383 443 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35081
I20260618 23:31:31.524029 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 23:31:31 dist-test-slave-6224 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825491, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 23:31:31 dist-test-slave-6224 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825491, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260618 23:31:31.550877 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:31.541932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.0.98.1:37517)
0618 23:31:31.542150 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:31.542156 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:31.542170 (+ 14us) server_negotiation.cc:407] Connection header received
0618 23:31:31.542212 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:31.542215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:31.542232 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:31.542320 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:31.543077 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:31.543984 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:31.544785 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:31.545174 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:31.545253 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:31.547719 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:31.547727 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:31.547727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:31.547736 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:31.549435 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:31.549812 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:31.549816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:31.549816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:31.549890 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:31.550181 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:31.550182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:31.550182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:31.550366 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:31.550518 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:31.550598 (+ 80us) server_negotiation.cc:299] Negotiation successful
0618 23:31:31.550695 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":100,"threads_started":1}
W20260618 23:31:31.551458 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34462
Jun 18 23:31:31 dist-test-slave-6224 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 23:31:31 dist-test-slave-6224 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260618 23:31:31.576478 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:31.568398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.0.98.1:37517)
0618 23:31:31.568489 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:31.568491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:31.568576 (+ 85us) server_negotiation.cc:407] Connection header received
0618 23:31:31.568643 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:31.568645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:31.568659 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:31.568726 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:31.569443 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:31.569958 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:31.570870 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:31.571076 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:31.571138 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:31.573792 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:31.573793 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:31.573794 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:31.573801 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:31.575270 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:31.575776 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:31.575777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:31.575777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:31.575830 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:31.576024 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:31.576024 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:31.576025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:31.576123 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:31.576221 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:31.576321 (+ 100us) server_negotiation.cc:299] Negotiation successful
0618 23:31:31.576405 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 23:31:31.577342 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 506
I20260618 23:31:31.584800 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 640
I20260618 23:31:31.591251 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 774
I20260618 23:31:31.596032 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 412
2026-06-18T23:31:31Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1992 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:31:31 dist-test-slave-6224 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:31:31 dist-test-slave-6224 krb5kdc[917](info): set up 2 sockets
Jun 18 23:31:31 dist-test-slave-6224 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260618 23:31:32.647341 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:31:32 dist-test-slave-6224 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:31:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:31:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:31:32.783058 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35077
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:32.865540 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:32.865782 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:32.865808 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:32.867045 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:31:32.867103 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:32.867118 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:32.867130 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:31:32.867142 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:31:32.868731 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:32.869033 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:32.869295 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:32.874742 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:32.874742 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:32.874778 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:32.874980 930 server_base.cc:1034] running on GCE node
I20260618 23:31:32.875375 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:32.875763 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:32.877012 930 hybrid_clock.cc:648] HybridClock initialized: now 1781825492876994 us; error 61 us; skew 500 ppm
Jun 18 23:31:32 dist-test-slave-6224 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825492, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:32.880097 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:31:32.880651 930 webserver.cc:460] Webserver started at http://127.0.98.62:41569/ using document root <none> and password file <none>
I20260618 23:31:32.880911 930 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:32.880985 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:32.881141 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:32.882274 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "2e911f008a454b0a93b4a711e8fe3d7f"
format_stamp: "Formatted at 2026-06-18 23:31:32 on dist-test-slave-6224"
server_key: "ea7728c0c9df866382a049b16be4c8e9"
server_key_iv: "915abd87de7ffb3e0f01a5bc871d9834"
server_key_version: "encryptionkey@0"
I20260618 23:31:32.882664 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e911f008a454b0a93b4a711e8fe3d7f"
format_stamp: "Formatted at 2026-06-18 23:31:32 on dist-test-slave-6224"
server_key: "ea7728c0c9df866382a049b16be4c8e9"
server_key_iv: "915abd87de7ffb3e0f01a5bc871d9834"
server_key_version: "encryptionkey@0"
I20260618 23:31:32.884411 930 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260618 23:31:32.885460 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:32.885751 930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:32.885871 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "2e911f008a454b0a93b4a711e8fe3d7f"
format_stamp: "Formatted at 2026-06-18 23:31:32 on dist-test-slave-6224"
server_key: "ea7728c0c9df866382a049b16be4c8e9"
server_key_iv: "915abd87de7ffb3e0f01a5bc871d9834"
server_key_version: "encryptionkey@0"
I20260618 23:31:32.885990 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:32.901779 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:32.904136 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:32.904310 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:32.908936 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35077
I20260618 23:31:32.908983 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35077 every 8 connection(s)
I20260618 23:31:32.909363 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:31:32.910475 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:32.913445 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f: Bootstrap starting.
I20260618 23:31:32.914211 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:32.914564 998 log.cc:824] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:32.915470 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f: No bootstrap required, opened a new log
I20260618 23:31:32.917366 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e911f008a454b0a93b4a711e8fe3d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35077 } }
I20260618 23:31:32.917567 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:32.917619 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e911f008a454b0a93b4a711e8fe3d7f, State: Initialized, Role: FOLLOWER
I20260618 23:31:32.917791 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e911f008a454b0a93b4a711e8fe3d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35077 } }
I20260618 23:31:32.917883 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:31:32.917936 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:31:32.917992 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:32.918681 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e911f008a454b0a93b4a711e8fe3d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35077 } }
I20260618 23:31:32.918825 998 leader_election.cc:302] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e911f008a454b0a93b4a711e8fe3d7f; no voters:
I20260618 23:31:32.919091 998 leader_election.cc:288] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:31:32.919126 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:32.919126 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 930
I20260618 23:31:32.919245 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:31:32.919426 392 external_mini_cluster.cc:1428] Setting key c05d02eae3f5ac49a88a639b41cee2c3
I20260618 23:31:32.919507 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:31:32.919515 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f [term 1 LEADER]: Becoming Leader. State: Replica: 2e911f008a454b0a93b4a711e8fe3d7f, State: Running, Role: LEADER
I20260618 23:31:32.919673 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e911f008a454b0a93b4a711e8fe3d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35077 } }
I20260618 23:31:32.920295 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e911f008a454b0a93b4a711e8fe3d7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e911f008a454b0a93b4a711e8fe3d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35077 } } }
I20260618 23:31:32.920421 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:32.920576 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e911f008a454b0a93b4a711e8fe3d7f. Latest consensus state: current_term: 1 leader_uuid: "2e911f008a454b0a93b4a711e8fe3d7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e911f008a454b0a93b4a711e8fe3d7f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35077 } } }
I20260618 23:31:32.920660 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:32.920766 1011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:31:32.921415 1011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:31:32.926721 1011 catalog_manager.cc:1347] Generated new cluster ID: f9e2143caeed4ea292cc84d1909065dc
I20260618 23:31:32.926810 1011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 18 23:31:32 dist-test-slave-6224 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:32.931085 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:32.920150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39222 (local address 127.0.98.62:35077)
0618 23:31:32.920375 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:32.920386 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:32.920427 (+ 41us) server_negotiation.cc:407] Connection header received
0618 23:31:32.920592 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:32.920601 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:32.920683 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:32.921690 (+ 1007us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:32.921999 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:32.923015 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:32.923946 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:32.924441 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:32.924562 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:32.927404 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:32.927408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:32.927409 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:32.927428 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:32.929277 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:32.929664 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:32.929666 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:32.929667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:32.929760 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:32.929985 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:32.929986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:32.929986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:32.930248 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:32.930439 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:32.930510 (+ 71us) server_negotiation.cc:299] Negotiation successful
0618 23:31:32.930668 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":97,"threads_started":1}
I20260618 23:31:32.944269 1011 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:31:32.945133 1011 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:31:32.950398 1011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e911f008a454b0a93b4a711e8fe3d7f: Generated new TSK 0
I20260618 23:31:32.950764 1011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260618 23:31:32.983883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35077
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:33.084087 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:33.084399 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:33.084448 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:33.085935 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:33.086021 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:33.086076 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:31:33.087898 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35077
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:33.088228 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:33.088513 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:33.094605 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:33.094707 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:33.094623 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:33.094933 1024 server_base.cc:1034] running on GCE node
I20260618 23:31:33.095242 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:33.095615 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:33.096868 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781825493096832 us; error 43 us; skew 500 ppm
Jun 18 23:31:33 dist-test-slave-6224 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:33.100178 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:31:33.100834 1024 webserver.cc:460] Webserver started at http://127.0.98.1:39399/ using document root <none> and password file <none>
I20260618 23:31:33.101122 1024 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:33.101198 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:33.101382 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:33.102733 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "9c907f578d6b43ada70cfc17a4e30e36"
format_stamp: "Formatted at 2026-06-18 23:31:33 on dist-test-slave-6224"
server_key: "e5a593b40da8ea7b08074b7ba4261207"
server_key_iv: "b3d772fc03a8f0e66aff9872457a2e51"
server_key_version: "encryptionkey@0"
I20260618 23:31:33.103206 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9c907f578d6b43ada70cfc17a4e30e36"
format_stamp: "Formatted at 2026-06-18 23:31:33 on dist-test-slave-6224"
server_key: "e5a593b40da8ea7b08074b7ba4261207"
server_key_iv: "b3d772fc03a8f0e66aff9872457a2e51"
server_key_version: "encryptionkey@0"
I20260618 23:31:33.104956 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260618 23:31:33.105978 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:33.106338 1024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:33.106500 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "9c907f578d6b43ada70cfc17a4e30e36"
format_stamp: "Formatted at 2026-06-18 23:31:33 on dist-test-slave-6224"
server_key: "e5a593b40da8ea7b08074b7ba4261207"
server_key_iv: "b3d772fc03a8f0e66aff9872457a2e51"
server_key_version: "encryptionkey@0"
I20260618 23:31:33.106621 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:33.124575 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:33.127391 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:33.127641 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:33.127951 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:33.128330 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:33.128391 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:33.128441 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:33.128465 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:33.136240 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42909
I20260618 23:31:33.136317 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42909 every 8 connection(s)
I20260618 23:31:33.136770 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:31:33.140472 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 1024
I20260618 23:31:33.140583 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:31:33.140734 392 external_mini_cluster.cc:1428] Setting key cf8fb99e2782c051222d61518e0c382d
Jun 18 23:31:33 dist-test-slave-6224 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:33.149422 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:33.137762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60267 (local address 127.0.98.62:35077)
0618 23:31:33.137813 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:33.137816 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:33.138080 (+ 264us) server_negotiation.cc:407] Connection header received
0618 23:31:33.138713 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:33.138714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:33.138724 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:33.138825 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:33.140051 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.140833 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.142241 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.142602 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.142688 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:33.145698 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:33.145701 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:33.145701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:33.145717 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:33.147648 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:33.148118 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:33.148120 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:33.148120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:33.148194 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:33.148544 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:33.148546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:33.148546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:33.148721 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:33.148816 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:33.149183 (+ 367us) server_negotiation.cc:299] Negotiation successful
0618 23:31:33.149290 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260618 23:31:33.149492 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:33.137716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35077 (local address 127.0.98.1:60267)
0618 23:31:33.137903 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:33.137955 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:33.138601 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:33.138909 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:33.138914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:33.139683 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:33.139936 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:33.139940 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.140986 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.140988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:33.142133 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:33.142136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.142738 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.142739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:33.142928 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:33.143699 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:33.143710 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:33.145529 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:33.147759 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:33.147766 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:33.147768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:33.148029 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:33.148309 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:33.148310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:33.148310 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:33.148445 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:33.148837 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:33.148844 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:33.149007 (+ 163us) client_negotiation.cc:769] Sending connection context
0618 23:31:33.149122 (+ 115us) client_negotiation.cc:240] Negotiation successful
0618 23:31:33.149294 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
I20260618 23:31:33.150321 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35077
I20260618 23:31:33.150439 1153 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:33.150656 1153 heartbeater.cc:511] Master 127.0.98.62:35077 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260618 23:31:33.151391 962 ts_manager.cc:194] Registered new tserver with Master: 9c907f578d6b43ada70cfc17a4e30e36 (127.0.98.1:42909)
I20260618 23:31:33.152115 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260618 23:31:33.175491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35077
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:33.276036 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:33.276340 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:33.276387 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:33.277835 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:33.277921 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:33.277976 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:31:33.279762 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35077
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:33.280098 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:33.280388 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:33.286494 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:33.286496 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:33.286522 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:33.286758 1158 server_base.cc:1034] running on GCE node
I20260618 23:31:33.287031 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:33.287302 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:33.288451 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781825493288413 us; error 68 us; skew 500 ppm
Jun 18 23:31:33 dist-test-slave-6224 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:33.291743 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:31:33.292369 1158 webserver.cc:460] Webserver started at http://127.0.98.2:43847/ using document root <none> and password file <none>
I20260618 23:31:33.292668 1158 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:33.292723 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:33.292879 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:33.294189 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "65da7f569e544b6fa9b1261037cce289"
format_stamp: "Formatted at 2026-06-18 23:31:33 on dist-test-slave-6224"
server_key: "ed376abd460b0805b81617d65d9d4eb2"
server_key_iv: "6318bbc5b38de2f4208b37b3b0669980"
server_key_version: "encryptionkey@0"
I20260618 23:31:33.294622 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "65da7f569e544b6fa9b1261037cce289"
format_stamp: "Formatted at 2026-06-18 23:31:33 on dist-test-slave-6224"
server_key: "ed376abd460b0805b81617d65d9d4eb2"
server_key_iv: "6318bbc5b38de2f4208b37b3b0669980"
server_key_version: "encryptionkey@0"
I20260618 23:31:33.296408 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260618 23:31:33.297465 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:33.297803 1158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:33.297925 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "65da7f569e544b6fa9b1261037cce289"
format_stamp: "Formatted at 2026-06-18 23:31:33 on dist-test-slave-6224"
server_key: "ed376abd460b0805b81617d65d9d4eb2"
server_key_iv: "6318bbc5b38de2f4208b37b3b0669980"
server_key_version: "encryptionkey@0"
I20260618 23:31:33.298019 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:33.310674 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:33.313498 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:33.313663 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:33.313915 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:33.314332 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:33.314373 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:33.314399 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:33.314451 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:33.322954 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43173
I20260618 23:31:33.323065 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43173 every 8 connection(s)
I20260618 23:31:33.323508 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
Jun 18 23:31:33 dist-test-slave-6224 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:33.332129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 1158
I20260618 23:31:33.332273 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:31:33.332494 392 external_mini_cluster.cc:1428] Setting key c71d40976c21222f923c3dfc77b76498
I20260618 23:31:33.335171 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:33.324670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54439 (local address 127.0.98.62:35077)
0618 23:31:33.324759 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:33.324762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:33.325077 (+ 315us) server_negotiation.cc:407] Connection header received
0618 23:31:33.325621 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:33.325623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:33.325637 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:33.325715 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:33.326833 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.327839 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.328947 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.329196 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.329283 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:33.332085 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:33.332087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:33.332087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:33.332096 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:33.333758 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:33.334163 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:33.334165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:33.334165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:33.334225 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:33.334502 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:33.334503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:33.334504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:33.334630 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:33.334705 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:33.334981 (+ 276us) server_negotiation.cc:299] Negotiation successful
0618 23:31:33.335075 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 23:31:33.335368 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:33.324657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35077 (local address 127.0.98.2:54439)
0618 23:31:33.324871 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:33.324922 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:33.325487 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:33.325814 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:33.325819 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:33.326482 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:33.326707 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:33.326711 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.327954 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.327955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:33.328832 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:33.328834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.329352 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.329353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:33.329469 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:33.330077 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:33.330085 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:33.331952 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:33.333857 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:33.333864 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:33.333866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:33.334085 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:33.334309 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:33.334310 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:33.334311 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:33.334424 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:33.334696 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:33.334702 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:33.334849 (+ 147us) client_negotiation.cc:769] Sending connection context
0618 23:31:33.334963 (+ 114us) client_negotiation.cc:240] Negotiation successful
0618 23:31:33.335120 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":114,"threads_started":1}
I20260618 23:31:33.335844 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35077
I20260618 23:31:33.335950 1287 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:33.336156 1287 heartbeater.cc:511] Master 127.0.98.62:35077 requested a full tablet report, sending...
I20260618 23:31:33.336642 962 ts_manager.cc:194] Registered new tserver with Master: 65da7f569e544b6fa9b1261037cce289 (127.0.98.2:43173)
I20260618 23:31:33.337177 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54439
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260618 23:31:33.359017 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35077
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:33.446352 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:33.446605 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:33.446645 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:33.447866 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:33.447937 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:33.447986 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:31:33.449442 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35077
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:33.449714 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:33.449956 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:33.455423 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:33.455415 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:33.455811 1292 server_base.cc:1034] running on GCE node
W20260618 23:31:33.455423 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:33.456197 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:33.456601 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:33.457842 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781825493457811 us; error 38 us; skew 500 ppm
Jun 18 23:31:33 dist-test-slave-6224 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:33.461004 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:31:33.461583 1292 webserver.cc:460] Webserver started at http://127.0.98.3:46499/ using document root <none> and password file <none>
I20260618 23:31:33.461853 1292 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:33.461930 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:33.462091 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:33.463347 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "5953ff0d78eb45c0b05b7023a6e8ee61"
format_stamp: "Formatted at 2026-06-18 23:31:33 on dist-test-slave-6224"
server_key: "d238dfc83273b121e58235d1ba820027"
server_key_iv: "8c07b7fbf1b2ce2a4da422f099b24926"
server_key_version: "encryptionkey@0"
I20260618 23:31:33.463793 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5953ff0d78eb45c0b05b7023a6e8ee61"
format_stamp: "Formatted at 2026-06-18 23:31:33 on dist-test-slave-6224"
server_key: "d238dfc83273b121e58235d1ba820027"
server_key_iv: "8c07b7fbf1b2ce2a4da422f099b24926"
server_key_version: "encryptionkey@0"
I20260618 23:31:33.466112 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260618 23:31:33.467110 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:33.467386 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:33.467505 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "5953ff0d78eb45c0b05b7023a6e8ee61"
format_stamp: "Formatted at 2026-06-18 23:31:33 on dist-test-slave-6224"
server_key: "d238dfc83273b121e58235d1ba820027"
server_key_iv: "8c07b7fbf1b2ce2a4da422f099b24926"
server_key_version: "encryptionkey@0"
I20260618 23:31:33.467643 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:33.502916 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:33.505687 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:33.505846 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:33.506122 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:33.506558 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:33.506600 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:33.506626 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:33.506668 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:33.514688 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33269
I20260618 23:31:33.514815 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33269 every 8 connection(s)
I20260618 23:31:33.515196 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:31:33.515827 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 1292
I20260618 23:31:33.515918 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:31:33.516150 392 external_mini_cluster.cc:1428] Setting key f812f5e218599b0bcfa81ffb90a82a0d
Jun 18 23:31:33 dist-test-slave-6224 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825493, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:33.526161 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:33.516172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59461 (local address 127.0.98.62:35077)
0618 23:31:33.516454 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:33.516457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:33.516707 (+ 250us) server_negotiation.cc:407] Connection header received
0618 23:31:33.517560 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:33.517561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:33.517574 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:33.517626 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:33.518646 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.519161 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.520300 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.520520 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.520609 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:33.523255 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:33.523256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:33.523257 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:33.523263 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:33.524698 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:33.525190 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:33.525191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:33.525191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:33.525243 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:33.525518 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:33.525519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:33.525519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:33.525624 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:33.525683 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:33.526023 (+ 340us) server_negotiation.cc:299] Negotiation successful
0618 23:31:33.526092 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245}
I20260618 23:31:33.526505 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:33.516129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35077 (local address 127.0.98.3:59461)
0618 23:31:33.516564 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:33.516609 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:33.517381 (+ 772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:33.517705 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:33.517709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:33.518334 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:33.518543 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:33.518547 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.519256 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.519257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:33.520210 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:33.520212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.520619 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.520620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:33.520775 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:33.521453 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:33.521462 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:33.523138 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:33.524795 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:33.524802 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:33.524805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:33.525105 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:33.525344 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:33.525344 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:33.525345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:33.525453 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:33.525690 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:33.525694 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:33.525860 (+ 166us) client_negotiation.cc:769] Sending connection context
0618 23:31:33.525961 (+ 101us) client_negotiation.cc:240] Negotiation successful
0618 23:31:33.526117 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":143,"threads_started":1}
I20260618 23:31:33.527289 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35077
I20260618 23:31:33.527419 1421 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:33.527667 1421 heartbeater.cc:511] Master 127.0.98.62:35077 requested a full tablet report, sending...
I20260618 23:31:33.528136 962 ts_manager.cc:194] Registered new tserver with Master: 5953ff0d78eb45c0b05b7023a6e8ee61 (127.0.98.3:33269)
I20260618 23:31:33.528654 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59461
I20260618 23:31:33.528775 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:31:33.537739 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:33.530730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39228 (local address 127.0.98.62:35077)
0618 23:31:33.530828 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:33.530831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:33.530889 (+ 58us) server_negotiation.cc:407] Connection header received
0618 23:31:33.530959 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:33.530960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:33.530970 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:33.531046 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:33.531636 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.532195 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.533159 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.533374 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.533446 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:33.534817 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:33.534818 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:33.534819 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:33.534826 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:33.536427 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:33.536838 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:33.536839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:33.536840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:33.536917 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:33.537114 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:33.537115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:33.537115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:33.537294 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:33.537388 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:33.537569 (+ 181us) server_negotiation.cc:299] Negotiation successful
0618 23:31:33.537659 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260618 23:31:33.540717 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:31:33.542694 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:31:33.551021 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:33.546304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47004 (local address 127.0.98.3:33269)
0618 23:31:33.546509 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:33.546516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:33.546540 (+ 24us) server_negotiation.cc:407] Connection header received
0618 23:31:33.546595 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:33.546598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:33.546616 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:33.546728 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:33.547109 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.548172 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.549665 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.550497 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.550593 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:33.550664 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:33.550860 (+ 196us) server_negotiation.cc:299] Negotiation successful
0618 23:31:33.550910 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":117,"threads_started":1}
I20260618 23:31:33.551014 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:33.546024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33269 (local address 127.0.0.1:47004)
0618 23:31:33.546351 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:33.546399 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:33.546486 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:33.546750 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:33.546754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:33.546758 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:33.546933 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:33.546937 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.548305 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.548307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:33.549544 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:33.549546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.550591 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.550592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:33.550713 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:33.550730 (+ 17us) client_negotiation.cc:769] Sending connection context
0618 23:31:33.550839 (+ 109us) client_negotiation.cc:240] Negotiation successful
0618 23:31:33.550925 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":133,"threads_started":1}
I20260618 23:31:33.551571 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:33.546351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51440 (local address 127.0.98.2:43173)
0618 23:31:33.546577 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:33.546583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:33.546876 (+ 293us) server_negotiation.cc:407] Connection header received
0618 23:31:33.547054 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:33.547059 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:33.547077 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:33.547136 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:33.547408 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.548680 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.549992 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.550716 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.550795 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:33.550861 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:33.551404 (+ 543us) server_negotiation.cc:299] Negotiation successful
0618 23:31:33.551456 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":100,"threads_started":1}
I20260618 23:31:33.551805 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:33.546033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43173 (local address 127.0.0.1:51440)
0618 23:31:33.546777 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:33.546790 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:33.546887 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:33.547258 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:33.547259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:33.547264 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:33.547322 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:33.547324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.548823 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.548824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:33.549889 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:33.549891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.550902 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.550904 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:33.550990 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:33.550997 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 23:31:33.551050 (+ 53us) client_negotiation.cc:240] Negotiation successful
0618 23:31:33.551107 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":75,"threads_started":1}
I20260618 23:31:33.551865 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:33.546562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43418 (local address 127.0.98.1:42909)
0618 23:31:33.546784 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:33.546789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:33.547523 (+ 734us) server_negotiation.cc:407] Connection header received
0618 23:31:33.547608 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:33.547611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:33.547629 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:33.547735 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:33.547886 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.549011 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.550332 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.551272 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.551341 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:33.551403 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:33.551719 (+ 316us) server_negotiation.cc:299] Negotiation successful
0618 23:31:33.551773 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":102,"threads_started":1}
I20260618 23:31:33.552074 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 824534f1ed5f40c3a3cf17eae469e84f (DEFAULT_TABLE table=test-table [id=ea2eee77f4da435184fac8e86b1f19b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:33.552251 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:33.546024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42909 (local address 127.0.0.1:43418)
0618 23:31:33.547434 (+ 1410us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:33.547445 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:33.547504 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:33.547755 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:33.547756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:33.547758 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:33.547808 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:33.547810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.549146 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.549147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:33.550214 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:33.550218 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.551442 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.551443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:33.551523 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:33.551527 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:31:33.551606 (+ 79us) client_negotiation.cc:240] Negotiation successful
0618 23:31:33.551668 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1391,"mutex_wait_us":23,"thread_start_us":120,"threads_started":1}
I20260618 23:31:33.552552 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824534f1ed5f40c3a3cf17eae469e84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:33.552714 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 824534f1ed5f40c3a3cf17eae469e84f (DEFAULT_TABLE table=test-table [id=ea2eee77f4da435184fac8e86b1f19b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:33.552965 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 824534f1ed5f40c3a3cf17eae469e84f (DEFAULT_TABLE table=test-table [id=ea2eee77f4da435184fac8e86b1f19b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:33.553133 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824534f1ed5f40c3a3cf17eae469e84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:33.553347 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824534f1ed5f40c3a3cf17eae469e84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:33.556195 1438 tablet_bootstrap.cc:492] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289: Bootstrap starting.
I20260618 23:31:33.556677 1439 tablet_bootstrap.cc:492] T 824534f1ed5f40c3a3cf17eae469e84f P 5953ff0d78eb45c0b05b7023a6e8ee61: Bootstrap starting.
I20260618 23:31:33.557026 1440 tablet_bootstrap.cc:492] T 824534f1ed5f40c3a3cf17eae469e84f P 9c907f578d6b43ada70cfc17a4e30e36: Bootstrap starting.
I20260618 23:31:33.557078 1438 tablet_bootstrap.cc:654] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:33.557416 1439 tablet_bootstrap.cc:654] T 824534f1ed5f40c3a3cf17eae469e84f P 5953ff0d78eb45c0b05b7023a6e8ee61: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:33.557839 1440 tablet_bootstrap.cc:654] T 824534f1ed5f40c3a3cf17eae469e84f P 9c907f578d6b43ada70cfc17a4e30e36: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:33.557909 1438 log.cc:824] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:33.558049 1439 log.cc:824] T 824534f1ed5f40c3a3cf17eae469e84f P 5953ff0d78eb45c0b05b7023a6e8ee61: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:33.558769 1440 log.cc:824] T 824534f1ed5f40c3a3cf17eae469e84f P 9c907f578d6b43ada70cfc17a4e30e36: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:33.559008 1438 tablet_bootstrap.cc:492] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289: No bootstrap required, opened a new log
I20260618 23:31:33.559090 1438 ts_tablet_manager.cc:1397] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 23:31:33.559098 1439 tablet_bootstrap.cc:492] T 824534f1ed5f40c3a3cf17eae469e84f P 5953ff0d78eb45c0b05b7023a6e8ee61: No bootstrap required, opened a new log
I20260618 23:31:33.559269 1439 ts_tablet_manager.cc:1397] T 824534f1ed5f40c3a3cf17eae469e84f P 5953ff0d78eb45c0b05b7023a6e8ee61: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 23:31:33.559875 1440 tablet_bootstrap.cc:492] T 824534f1ed5f40c3a3cf17eae469e84f P 9c907f578d6b43ada70cfc17a4e30e36: No bootstrap required, opened a new log
I20260618 23:31:33.559960 1440 ts_tablet_manager.cc:1397] T 824534f1ed5f40c3a3cf17eae469e84f P 9c907f578d6b43ada70cfc17a4e30e36: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 23:31:33.560879 1438 raft_consensus.cc:348] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5953ff0d78eb45c0b05b7023a6e8ee61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33269 } } peers { permanent_uuid: "9c907f578d6b43ada70cfc17a4e30e36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42909 } } peers { permanent_uuid: "65da7f569e544b6fa9b1261037cce289" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43173 } }
I20260618 23:31:33.561062 1438 raft_consensus.cc:374] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:33.561106 1438 raft_consensus.cc:729] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65da7f569e544b6fa9b1261037cce289, State: Initialized, Role: FOLLOWER
I20260618 23:31:33.561244 1438 consensus_queue.cc:260] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5953ff0d78eb45c0b05b7023a6e8ee61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33269 } } peers { permanent_uuid: "9c907f578d6b43ada70cfc17a4e30e36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42909 } } peers { permanent_uuid: "65da7f569e544b6fa9b1261037cce289" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43173 } }
I20260618 23:31:33.561566 1438 ts_tablet_manager.cc:1428] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260618 23:31:33.561643 1287 heartbeater.cc:503] Master 127.0.98.62:35077 was elected leader, sending a full tablet report...
I20260618 23:31:33.562180 1439 raft_consensus.cc:348] T 824534f1ed5f40c3a3cf17eae469e84f P 5953ff0d78eb45c0b05b7023a6e8ee61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5953ff0d78eb45c0b05b7023a6e8ee61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33269 } } peers { permanent_uuid: "9c907f578d6b43ada70cfc17a4e30e36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42909 } } peers { permanent_uuid: "65da7f569e544b6fa9b1261037cce289" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43173 } }
I20260618 23:31:33.563228 1439 raft_consensus.cc:374] T 824534f1ed5f40c3a3cf17eae469e84f P 5953ff0d78eb45c0b05b7023a6e8ee61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:33.563289 1439 raft_consensus.cc:729] T 824534f1ed5f40c3a3cf17eae469e84f P 5953ff0d78eb45c0b05b7023a6e8ee61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5953ff0d78eb45c0b05b7023a6e8ee61, State: Initialized, Role: FOLLOWER
I20260618 23:31:33.563441 1439 consensus_queue.cc:260] T 824534f1ed5f40c3a3cf17eae469e84f P 5953ff0d78eb45c0b05b7023a6e8ee61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5953ff0d78eb45c0b05b7023a6e8ee61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33269 } } peers { permanent_uuid: "9c907f578d6b43ada70cfc17a4e30e36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42909 } } peers { permanent_uuid: "65da7f569e544b6fa9b1261037cce289" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43173 } }
I20260618 23:31:33.563844 1439 ts_tablet_manager.cc:1428] T 824534f1ed5f40c3a3cf17eae469e84f P 5953ff0d78eb45c0b05b7023a6e8ee61: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260618 23:31:33.563868 1421 heartbeater.cc:503] Master 127.0.98.62:35077 was elected leader, sending a full tablet report...
I20260618 23:31:33.563815 1440 raft_consensus.cc:348] T 824534f1ed5f40c3a3cf17eae469e84f P 9c907f578d6b43ada70cfc17a4e30e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5953ff0d78eb45c0b05b7023a6e8ee61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33269 } } peers { permanent_uuid: "9c907f578d6b43ada70cfc17a4e30e36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42909 } } peers { permanent_uuid: "65da7f569e544b6fa9b1261037cce289" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43173 } }
I20260618 23:31:33.564014 1440 raft_consensus.cc:374] T 824534f1ed5f40c3a3cf17eae469e84f P 9c907f578d6b43ada70cfc17a4e30e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:33.564061 1440 raft_consensus.cc:729] T 824534f1ed5f40c3a3cf17eae469e84f P 9c907f578d6b43ada70cfc17a4e30e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c907f578d6b43ada70cfc17a4e30e36, State: Initialized, Role: FOLLOWER
I20260618 23:31:33.564209 1440 consensus_queue.cc:260] T 824534f1ed5f40c3a3cf17eae469e84f P 9c907f578d6b43ada70cfc17a4e30e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5953ff0d78eb45c0b05b7023a6e8ee61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33269 } } peers { permanent_uuid: "9c907f578d6b43ada70cfc17a4e30e36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42909 } } peers { permanent_uuid: "65da7f569e544b6fa9b1261037cce289" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43173 } }
I20260618 23:31:33.564579 1153 heartbeater.cc:503] Master 127.0.98.62:35077 was elected leader, sending a full tablet report...
I20260618 23:31:33.564869 1440 ts_tablet_manager.cc:1428] T 824534f1ed5f40c3a3cf17eae469e84f P 9c907f578d6b43ada70cfc17a4e30e36: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
W20260618 23:31:33.574433 1288 tablet.cc:2369] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:31:33.637701 1154 tablet.cc:2369] T 824534f1ed5f40c3a3cf17eae469e84f P 9c907f578d6b43ada70cfc17a4e30e36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:31:33.701845 1444 raft_consensus.cc:482] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:31:33.702019 1444 raft_consensus.cc:504] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5953ff0d78eb45c0b05b7023a6e8ee61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33269 } } peers { permanent_uuid: "9c907f578d6b43ada70cfc17a4e30e36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42909 } } peers { permanent_uuid: "65da7f569e544b6fa9b1261037cce289" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43173 } }
I20260618 23:31:33.702436 1444 leader_election.cc:288] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5953ff0d78eb45c0b05b7023a6e8ee61 (127.0.98.3:33269), 9c907f578d6b43ada70cfc17a4e30e36 (127.0.98.1:42909)
I20260618 23:31:33.706799 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:33.702621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42909 (local address 127.0.98.2:57263)
0618 23:31:33.702680 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:33.702693 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:33.702738 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:33.703289 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:33.703290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:33.703293 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:33.703383 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:33.703385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.704425 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.704426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:33.705613 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:33.705615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.706524 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.706525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:33.706594 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:33.706598 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:31:33.706656 (+ 58us) client_negotiation.cc:240] Negotiation successful
0618 23:31:33.706691 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260618 23:31:33.706846 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:33.703028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57263 (local address 127.0.98.1:42909)
0618 23:31:33.703140 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:33.703143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:33.703154 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:31:33.703188 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:33.703189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:33.703204 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:33.703263 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:33.703457 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.704222 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.705733 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.706393 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.706522 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:33.706537 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:33.706686 (+ 149us) server_negotiation.cc:299] Negotiation successful
0618 23:31:33.706747 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260618 23:31:33.707103 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:33.702621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33269 (local address 127.0.98.2:57981)
0618 23:31:33.703702 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:33.703715 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:33.703791 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:33.703966 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:33.703967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:33.703968 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:33.704033 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:33.704034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.704865 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.704865 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:33.705895 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:33.705896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.706765 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.706766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:33.706896 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:33.706900 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:31:33.707001 (+ 101us) client_negotiation.cc:240] Negotiation successful
0618 23:31:33.707036 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"mutex_wait_us":22,"thread_start_us":124,"threads_started":1}
I20260618 23:31:33.707165 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:33.702890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57981 (local address 127.0.98.3:33269)
0618 23:31:33.702940 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:33.702943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:33.703766 (+ 823us) server_negotiation.cc:407] Connection header received
0618 23:31:33.703872 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:33.703873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:33.703884 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:33.703911 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:33.704100 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.704758 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.705990 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.706659 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.706739 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:33.706750 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:33.707015 (+ 265us) server_negotiation.cc:299] Negotiation successful
0618 23:31:33.707087 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260618 23:31:33.707472 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824534f1ed5f40c3a3cf17eae469e84f" candidate_uuid: "65da7f569e544b6fa9b1261037cce289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5953ff0d78eb45c0b05b7023a6e8ee61" is_pre_election: true
I20260618 23:31:33.707476 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824534f1ed5f40c3a3cf17eae469e84f" candidate_uuid: "65da7f569e544b6fa9b1261037cce289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c907f578d6b43ada70cfc17a4e30e36" is_pre_election: true
I20260618 23:31:33.707676 1375 raft_consensus.cc:2454] T 824534f1ed5f40c3a3cf17eae469e84f P 5953ff0d78eb45c0b05b7023a6e8ee61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65da7f569e544b6fa9b1261037cce289 in term 0.
I20260618 23:31:33.707723 1107 raft_consensus.cc:2454] T 824534f1ed5f40c3a3cf17eae469e84f P 9c907f578d6b43ada70cfc17a4e30e36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65da7f569e544b6fa9b1261037cce289 in term 0.
I20260618 23:31:33.707942 1175 leader_election.cc:302] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5953ff0d78eb45c0b05b7023a6e8ee61, 65da7f569e544b6fa9b1261037cce289; no voters:
I20260618 23:31:33.708110 1444 raft_consensus.cc:2781] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:31:33.708176 1444 raft_consensus.cc:482] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:31:33.708213 1444 raft_consensus.cc:3037] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:33.709100 1444 raft_consensus.cc:504] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5953ff0d78eb45c0b05b7023a6e8ee61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33269 } } peers { permanent_uuid: "9c907f578d6b43ada70cfc17a4e30e36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42909 } } peers { permanent_uuid: "65da7f569e544b6fa9b1261037cce289" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43173 } }
I20260618 23:31:33.709281 1444 leader_election.cc:288] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289 [CANDIDATE]: Term 1 election: Requested vote from peers 5953ff0d78eb45c0b05b7023a6e8ee61 (127.0.98.3:33269), 9c907f578d6b43ada70cfc17a4e30e36 (127.0.98.1:42909)
I20260618 23:31:33.709494 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824534f1ed5f40c3a3cf17eae469e84f" candidate_uuid: "65da7f569e544b6fa9b1261037cce289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5953ff0d78eb45c0b05b7023a6e8ee61"
I20260618 23:31:33.709601 1375 raft_consensus.cc:3037] T 824534f1ed5f40c3a3cf17eae469e84f P 5953ff0d78eb45c0b05b7023a6e8ee61 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:33.709537 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824534f1ed5f40c3a3cf17eae469e84f" candidate_uuid: "65da7f569e544b6fa9b1261037cce289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c907f578d6b43ada70cfc17a4e30e36"
I20260618 23:31:33.709689 1107 raft_consensus.cc:3037] T 824534f1ed5f40c3a3cf17eae469e84f P 9c907f578d6b43ada70cfc17a4e30e36 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:33.710590 1375 raft_consensus.cc:2454] T 824534f1ed5f40c3a3cf17eae469e84f P 5953ff0d78eb45c0b05b7023a6e8ee61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65da7f569e544b6fa9b1261037cce289 in term 1.
I20260618 23:31:33.710728 1107 raft_consensus.cc:2454] T 824534f1ed5f40c3a3cf17eae469e84f P 9c907f578d6b43ada70cfc17a4e30e36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65da7f569e544b6fa9b1261037cce289 in term 1.
I20260618 23:31:33.710770 1175 leader_election.cc:302] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5953ff0d78eb45c0b05b7023a6e8ee61, 65da7f569e544b6fa9b1261037cce289; no voters:
I20260618 23:31:33.710888 1444 raft_consensus.cc:2781] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:33.711072 1444 raft_consensus.cc:686] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289 [term 1 LEADER]: Becoming Leader. State: Replica: 65da7f569e544b6fa9b1261037cce289, State: Running, Role: LEADER
I20260618 23:31:33.711179 1444 consensus_queue.cc:237] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5953ff0d78eb45c0b05b7023a6e8ee61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33269 } } peers { permanent_uuid: "9c907f578d6b43ada70cfc17a4e30e36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42909 } } peers { permanent_uuid: "65da7f569e544b6fa9b1261037cce289" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43173 } }
I20260618 23:31:33.712229 961 catalog_manager.cc:5495] T 824534f1ed5f40c3a3cf17eae469e84f P 65da7f569e544b6fa9b1261037cce289 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65da7f569e544b6fa9b1261037cce289 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "65da7f569e544b6fa9b1261037cce289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5953ff0d78eb45c0b05b7023a6e8ee61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c907f578d6b43ada70cfc17a4e30e36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65da7f569e544b6fa9b1261037cce289" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43173 } health_report { overall_health: HEALTHY } } }
Jun 18 23:31:33 dist-test-slave-6224 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260618 23:31:33.740108 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:33.731229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.0.98.1:42909)
0618 23:31:33.731323 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:33.731326 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:33.731442 (+ 116us) server_negotiation.cc:407] Connection header received
0618 23:31:33.731573 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:33.731574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:33.731587 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:33.731659 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:33.732310 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.732855 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.733658 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.733856 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.733965 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:33.736808 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:33.736824 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:33.736824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:33.736836 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:33.738536 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:33.739022 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:33.739025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:33.739025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:33.739116 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:33.739368 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:33.739369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:33.739370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:33.739536 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:33.739741 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:33.739879 (+ 138us) server_negotiation.cc:299] Negotiation successful
0618 23:31:33.739963 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Jun 18 23:31:33 dist-test-slave-6224 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825493, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 23:31:33 dist-test-slave-6224 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825493, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
W20260618 23:31:33.765941 1422 tablet.cc:2369] T 824534f1ed5f40c3a3cf17eae469e84f P 5953ff0d78eb45c0b05b7023a6e8ee61: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:31:33.767807 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:33.759623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.0.98.1:42909)
0618 23:31:33.759728 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:33.759731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:33.759882 (+ 151us) server_negotiation.cc:407] Connection header received
0618 23:31:33.759999 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:33.760000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:33.760011 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:33.760087 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:33.760713 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.761216 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.761977 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.762181 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.762269 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:33.764934 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:33.764935 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:33.764936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:33.764942 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:33.766522 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:33.766959 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:33.766960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:33.766960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:33.767014 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:33.767225 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:33.767226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:33.767226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:33.767343 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:33.767419 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:33.767675 (+ 256us) server_negotiation.cc:299] Negotiation successful
0618 23:31:33.767739 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Jun 18 23:31:33 dist-test-slave-6224 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 23:31:33 dist-test-slave-6224 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825493, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260618 23:31:33.794459 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:33.786297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43438 (local address 127.0.98.1:42909)
0618 23:31:33.786417 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:33.786420 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:33.786588 (+ 168us) server_negotiation.cc:407] Connection header received
0618 23:31:33.786651 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:33.786653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:33.786665 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:33.786737 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:33.787316 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.787912 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.788773 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:33.788961 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:33.789021 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:33.791846 (+ 2825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:33.791852 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:33.791853 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:33.791861 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:33.793337 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:33.793681 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:33.793689 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:33.793690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:33.793750 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:33.793944 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:33.793944 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:33.793945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:33.794050 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:33.794138 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:33.794253 (+ 115us) server_negotiation.cc:299] Negotiation successful
0618 23:31:33.794325 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260618 23:31:33.795048 1067 tablet_service.cc:2908] Rejecting scan request for tablet 824534f1ed5f40c3a3cf17eae469e84f: Uninitialized: clean time has not yet been initialized
I20260618 23:31:33.796150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 1024
I20260618 23:31:33.801690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 1158
I20260618 23:31:33.806730 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 1292
I20260618 23:31:33.811952 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 930
2026-06-18T23:31:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2216 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:31:33 dist-test-slave-6224 krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:31:33 dist-test-slave-6224 krb5kdc[1462](info): set up 2 sockets
Jun 18 23:31:33 dist-test-slave-6224 krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260618 23:31:34.855263 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:31:34 dist-test-slave-6224 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:31:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:31:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:31:34.996563 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33725
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:35.078279 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:35.078497 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:35.078519 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:35.079793 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:31:35.079844 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260618 23:31:35.079864 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:35.079877 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:35.079890 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:31:35.079903 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:31:35.081545 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33725
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:35.081807 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:35.082070 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:35.087435 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:35.087473 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:35.087635 1475 server_base.cc:1034] running on GCE node
W20260618 23:31:35.087464 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:35.088081 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:35.088424 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:35.089576 1475 hybrid_clock.cc:648] HybridClock initialized: now 1781825495089562 us; error 49 us; skew 500 ppm
Jun 18 23:31:35 dist-test-slave-6224 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:35.092502 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:31:35.093029 1475 webserver.cc:460] Webserver started at http://127.0.98.62:33681/ using document root <none> and password file <none>
I20260618 23:31:35.093290 1475 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:35.093343 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:35.093453 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:35.094697 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "75bae1580e4d450ab7fe5da074588771"
format_stamp: "Formatted at 2026-06-18 23:31:35 on dist-test-slave-6224"
server_key: "38cfc1a145f967d148518dfe794f0bcc"
server_key_iv: "5e3af164fd86a3182c1178f792d336c7"
server_key_version: "encryptionkey@0"
I20260618 23:31:35.095088 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "75bae1580e4d450ab7fe5da074588771"
format_stamp: "Formatted at 2026-06-18 23:31:35 on dist-test-slave-6224"
server_key: "38cfc1a145f967d148518dfe794f0bcc"
server_key_iv: "5e3af164fd86a3182c1178f792d336c7"
server_key_version: "encryptionkey@0"
I20260618 23:31:35.096952 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260618 23:31:35.097926 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:35.098181 1475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:35.098284 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "75bae1580e4d450ab7fe5da074588771"
format_stamp: "Formatted at 2026-06-18 23:31:35 on dist-test-slave-6224"
server_key: "38cfc1a145f967d148518dfe794f0bcc"
server_key_iv: "5e3af164fd86a3182c1178f792d336c7"
server_key_version: "encryptionkey@0"
I20260618 23:31:35.098373 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:35.113958 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:35.116420 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:35.116541 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:35.121532 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33725
I20260618 23:31:35.121737 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33725 every 8 connection(s)
I20260618 23:31:35.122020 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:31:35.122915 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:35.125702 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771: Bootstrap starting.
I20260618 23:31:35.126464 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:35.126809 1553 log.cc:824] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:35.127754 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771: No bootstrap required, opened a new log
I20260618 23:31:35.130064 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75bae1580e4d450ab7fe5da074588771" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33725 } }
I20260618 23:31:35.130259 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:35.130334 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75bae1580e4d450ab7fe5da074588771, State: Initialized, Role: FOLLOWER
I20260618 23:31:35.130528 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75bae1580e4d450ab7fe5da074588771" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33725 } }
I20260618 23:31:35.130620 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:31:35.130666 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:31:35.130722 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:35.131373 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75bae1580e4d450ab7fe5da074588771" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33725 } }
I20260618 23:31:35.131502 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75bae1580e4d450ab7fe5da074588771; no voters:
I20260618 23:31:35.131793 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:31:35.131780 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 1475
I20260618 23:31:35.131827 1556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:35.131907 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:31:35.131983 1556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771 [term 1 LEADER]: Becoming Leader. State: Replica: 75bae1580e4d450ab7fe5da074588771, State: Running, Role: LEADER
I20260618 23:31:35.132094 392 external_mini_cluster.cc:1428] Setting key 12e5eb8b6fd34dfb627ba7d4536521e6
I20260618 23:31:35.132076 1556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75bae1580e4d450ab7fe5da074588771" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33725 } }
I20260618 23:31:35.132108 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:31:35.134409 1558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75bae1580e4d450ab7fe5da074588771. Latest consensus state: current_term: 1 leader_uuid: "75bae1580e4d450ab7fe5da074588771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75bae1580e4d450ab7fe5da074588771" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33725 } } }
I20260618 23:31:35.134521 1558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771 [sys.catalog]: This master's current role is: LEADER
W20260618 23:31:35.134784 1572 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:31:35.134873 1572 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:31:35.135005 1557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75bae1580e4d450ab7fe5da074588771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75bae1580e4d450ab7fe5da074588771" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33725 } } }
I20260618 23:31:35.135119 1557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771 [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:35.135208 1574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:31:35.135397 1574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:31:35.137398 1574 catalog_manager.cc:1347] Generated new cluster ID: 0c57fa7b8e664c99b97f59133450a961
I20260618 23:31:35.137463 1574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 18 23:31:35 dist-test-slave-6224 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825494, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:35.144428 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.132696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45662 (local address 127.0.98.62:33725)
0618 23:31:35.132887 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:35.132898 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:35.132945 (+ 47us) server_negotiation.cc:407] Connection header received
0618 23:31:35.133072 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:35.133078 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:35.133138 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:35.133288 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:35.134053 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.135582 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.136513 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.137125 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.137258 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.140390 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:35.140395 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:35.140395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:35.140407 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:35.142539 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:35.143083 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:35.143086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:35.143087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:35.143199 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:35.143444 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:35.143445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:35.143445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:35.143773 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:35.143966 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:35.144079 (+ 113us) server_negotiation.cc:299] Negotiation successful
0618 23:31:35.144218 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260618 23:31:35.146221 1574 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:31:35.146906 1574 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:31:35.152312 1574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771: Generated new TSK 0
I20260618 23:31:35.152479 1574 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260618 23:31:35.181854 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33725
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:35.265105 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:35.265332 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:35.265370 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:35.266510 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:35.266574 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:35.266609 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 23:31:35.266655 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:31:35.268201 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:35.268473 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:35.268697 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:35.274016 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:35.274044 1579 server_base.cc:1034] running on GCE node
W20260618 23:31:35.274174 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:35.274065 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:35.274485 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:35.274751 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:35.275914 1579 hybrid_clock.cc:648] HybridClock initialized: now 1781825495275895 us; error 37 us; skew 500 ppm
Jun 18 23:31:35 dist-test-slave-6224 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:35.278767 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:31:35.279325 1579 webserver.cc:460] Webserver started at http://127.0.98.1:35593/ using document root <none> and password file <none>
I20260618 23:31:35.279639 1579 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:35.279711 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:35.279857 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:35.281000 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "7f1309bc848a44b1808e1e740c132d27"
format_stamp: "Formatted at 2026-06-18 23:31:35 on dist-test-slave-6224"
server_key: "81f6f6b314bed3992134b9bf24c28199"
server_key_iv: "aa3551e156470c37dc8771f1a737b493"
server_key_version: "encryptionkey@0"
I20260618 23:31:35.281385 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7f1309bc848a44b1808e1e740c132d27"
format_stamp: "Formatted at 2026-06-18 23:31:35 on dist-test-slave-6224"
server_key: "81f6f6b314bed3992134b9bf24c28199"
server_key_iv: "aa3551e156470c37dc8771f1a737b493"
server_key_version: "encryptionkey@0"
I20260618 23:31:35.282835 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:31:35.283727 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:35.283980 1579 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:31:35.284096 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "7f1309bc848a44b1808e1e740c132d27"
format_stamp: "Formatted at 2026-06-18 23:31:35 on dist-test-slave-6224"
server_key: "81f6f6b314bed3992134b9bf24c28199"
server_key_iv: "aa3551e156470c37dc8771f1a737b493"
server_key_version: "encryptionkey@0"
I20260618 23:31:35.284181 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:35.300838 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:35.303208 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:35.303387 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:35.304113 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:35.304165 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:35.304214 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:35.304241 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 23:31:35 dist-test-slave-6224 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:35.312199 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45933
I20260618 23:31:35.312315 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45933 every 8 connection(s)
I20260618 23:31:35.312638 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:31:35.315645 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.304713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60719 (local address 127.0.98.62:33725)
0618 23:31:35.304807 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:35.304811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:35.305144 (+ 333us) server_negotiation.cc:407] Connection header received
0618 23:31:35.305978 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:35.305980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:35.305995 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:35.306078 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:35.307030 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.307747 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.309069 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.309300 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.309385 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.311975 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:35.311977 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:35.311977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:35.311986 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:35.313810 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:35.314453 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:35.314454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:35.314455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:35.314520 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:35.314842 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:35.314843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:35.314843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:35.314962 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:35.315021 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:35.315408 (+ 387us) server_negotiation.cc:299] Negotiation successful
0618 23:31:35.315513 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 23:31:35.315879 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.304705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33725 (local address 127.0.98.1:60719)
0618 23:31:35.304965 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:35.305021 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:35.305815 (+ 794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:35.306179 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:35.306184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:35.306590 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:35.306912 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.306917 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.307863 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.307865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.308963 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.308967 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.309409 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.309411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.309575 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.310202 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:35.310221 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:35.311858 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:35.313941 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:35.313949 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:35.313951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:35.314345 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:35.314621 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:35.314622 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:35.314622 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:35.314765 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:35.315066 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:35.315072 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:35.315231 (+ 159us) client_negotiation.cc:769] Sending connection context
0618 23:31:35.315355 (+ 124us) client_negotiation.cc:240] Negotiation successful
0618 23:31:35.315531 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":135,"threads_started":1}
I20260618 23:31:35.316568 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33725
I20260618 23:31:35.316706 1710 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:35.316937 1710 heartbeater.cc:511] Master 127.0.98.62:33725 requested a full tablet report, sending...
I20260618 23:31:35.317651 1506 ts_manager.cc:194] Registered new tserver with Master: 7f1309bc848a44b1808e1e740c132d27 (127.0.98.1:45933)
I20260618 23:31:35.318368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 1579
I20260618 23:31:35.318495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:31:35.318637 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60719
I20260618 23:31:35.318712 392 external_mini_cluster.cc:1428] Setting key abdcdc993e94f9b30b1e93950ee8abb3
I20260618 23:31:35.324905 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.317785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42837 (local address 127.0.98.62:33725)
0618 23:31:35.317836 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:35.317838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:35.318127 (+ 289us) server_negotiation.cc:407] Connection header received
0618 23:31:35.318182 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:35.318183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:35.318197 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:35.318262 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:35.318662 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.319681 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.320474 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.320695 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.320784 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.321921 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:35.321922 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:35.321923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:35.321930 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:35.323635 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:35.324025 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:35.324026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:35.324027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:35.324101 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:35.324324 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:35.324324 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:35.324325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:35.324457 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:35.324534 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:35.324699 (+ 165us) server_negotiation.cc:299] Negotiation successful
0618 23:31:35.324799 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":188,"server-negotiator.queue_time_us":31}
I20260618 23:31:35.324918 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.317830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33725 (local address 127.0.98.1:42837)
0618 23:31:35.318023 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:35.318037 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:35.318116 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:35.318280 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:35.318281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:35.318506 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:35.318574 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.318577 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.319795 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.319798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.320373 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.320375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.320782 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.320783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.320857 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.321421 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:35.321426 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:35.321797 (+ 371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:35.323737 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:35.323738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:35.323739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:35.323937 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:35.324170 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:35.324170 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:35.324171 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:35.324258 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:35.324524 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:35.324525 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:35.324584 (+ 59us) client_negotiation.cc:769] Sending connection context
0618 23:31:35.324675 (+ 91us) client_negotiation.cc:240] Negotiation successful
0618 23:31:35.324796 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":100,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260618 23:31:35.347157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33725
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:35.429849 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:35.430080 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:35.430105 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:35.431331 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:35.431384 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:35.431406 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 23:31:35.431423 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:31:35.433269 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:35.433521 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:35.433764 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:35.439231 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:35.439287 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:35.439231 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:35.439420 1719 server_base.cc:1034] running on GCE node
I20260618 23:31:35.439860 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:35.440135 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:35.441313 1719 hybrid_clock.cc:648] HybridClock initialized: now 1781825495441281 us; error 65 us; skew 500 ppm
Jun 18 23:31:35 dist-test-slave-6224 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:35.444259 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:31:35.444784 1719 webserver.cc:460] Webserver started at http://127.0.98.2:35297/ using document root <none> and password file <none>
I20260618 23:31:35.445063 1719 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:35.445137 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:35.445290 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:35.446352 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "233d75d48e4b465381298a1f2e2ff5d2"
format_stamp: "Formatted at 2026-06-18 23:31:35 on dist-test-slave-6224"
server_key: "f598e35e0a59c8ba93fad856c5406af8"
server_key_iv: "d2d0fd6f0302ee1374ba3c6983470a8f"
server_key_version: "encryptionkey@0"
I20260618 23:31:35.446733 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "233d75d48e4b465381298a1f2e2ff5d2"
format_stamp: "Formatted at 2026-06-18 23:31:35 on dist-test-slave-6224"
server_key: "f598e35e0a59c8ba93fad856c5406af8"
server_key_iv: "d2d0fd6f0302ee1374ba3c6983470a8f"
server_key_version: "encryptionkey@0"
I20260618 23:31:35.448338 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:31:35.449254 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:35.449553 1719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:35.449661 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "233d75d48e4b465381298a1f2e2ff5d2"
format_stamp: "Formatted at 2026-06-18 23:31:35 on dist-test-slave-6224"
server_key: "f598e35e0a59c8ba93fad856c5406af8"
server_key_iv: "d2d0fd6f0302ee1374ba3c6983470a8f"
server_key_version: "encryptionkey@0"
I20260618 23:31:35.449745 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:35.462836 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:35.465185 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:35.465306 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:35.465862 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:35.465930 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:35.465963 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:35.465986 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 23:31:35 dist-test-slave-6224 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:35.474129 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38663
I20260618 23:31:35.474613 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:31:35.474651 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38663 every 8 connection(s)
I20260618 23:31:35.477389 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.466497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37493 (local address 127.0.98.62:33725)
0618 23:31:35.466591 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:35.466594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:35.466877 (+ 283us) server_negotiation.cc:407] Connection header received
0618 23:31:35.467582 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:35.467583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:35.467598 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:35.467689 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:35.468482 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.469024 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.470298 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.470510 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.470547 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.473854 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:35.473856 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:35.473857 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:35.473866 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:35.475780 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:35.476233 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:35.476234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:35.476234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:35.476303 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:35.476665 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:35.476666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:35.476667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:35.476832 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:35.476907 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:35.477220 (+ 313us) server_negotiation.cc:299] Negotiation successful
0618 23:31:35.477311 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 23:31:35.477470 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.466473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33725 (local address 127.0.98.2:37493)
0618 23:31:35.466700 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:35.466750 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:35.467423 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:35.467786 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:35.467793 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:35.468127 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:35.468359 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.468363 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.469168 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.469170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.470178 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.470181 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.470653 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.470654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.470775 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.471538 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:35.471582 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:35.473702 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:35.475897 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:35.475904 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:35.475906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:35.476131 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:35.476463 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:35.476463 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:35.476464 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:35.476562 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:35.476917 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:35.476921 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:35.477057 (+ 136us) client_negotiation.cc:769] Sending connection context
0618 23:31:35.477153 (+ 96us) client_negotiation.cc:240] Negotiation successful
0618 23:31:35.477289 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":135,"threads_started":1}
I20260618 23:31:35.478219 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33725
I20260618 23:31:35.478328 1850 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:35.478554 1850 heartbeater.cc:511] Master 127.0.98.62:33725 requested a full tablet report, sending...
I20260618 23:31:35.479048 1507 ts_manager.cc:194] Registered new tserver with Master: 233d75d48e4b465381298a1f2e2ff5d2 (127.0.98.2:38663)
I20260618 23:31:35.479666 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37493
I20260618 23:31:35.483366 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 1719
I20260618 23:31:35.483482 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:31:35.483717 392 external_mini_cluster.cc:1428] Setting key dfb2c9742073e290b9d0f27cef6a40d2
I20260618 23:31:35.486320 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.479399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33725 (local address 127.0.98.2:48023)
0618 23:31:35.479694 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:35.479706 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:35.479789 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:35.479968 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:35.479969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:35.480152 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:35.480234 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.480235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.480940 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.480941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.481507 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.481508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.481869 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.481870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.481938 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.482484 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:35.482488 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:35.482771 (+ 283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:35.485024 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:35.485025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:35.485025 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:35.485269 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:35.485522 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:35.485523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:35.485523 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:35.485597 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:35.485947 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:35.485948 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:35.485997 (+ 49us) client_negotiation.cc:769] Sending connection context
0618 23:31:35.486084 (+ 87us) client_negotiation.cc:240] Negotiation successful
0618 23:31:35.486216 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":125,"threads_started":1}
I20260618 23:31:35.486385 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.479478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48023 (local address 127.0.98.62:33725)
0618 23:31:35.479614 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:35.479616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:35.479770 (+ 154us) server_negotiation.cc:407] Connection header received
0618 23:31:35.479855 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:35.479857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:35.479869 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:35.479941 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:35.480330 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.480842 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.481594 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.481783 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.481869 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.482865 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:35.482866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:35.482867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:35.482874 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:35.484930 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:35.485370 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:35.485371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:35.485371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:35.485443 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:35.485681 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:35.485682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:35.485682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:35.485870 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:35.485931 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:35.486135 (+ 204us) server_negotiation.cc:299] Negotiation successful
0618 23:31:35.486248 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260618 23:31:35.510430 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33725
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:35.593528 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:35.593757 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:35.593788 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:35.594918 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:35.594959 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:35.594980 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 23:31:35.594996 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:31:35.596839 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:33725
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:35.597041 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:35.597221 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:35.602864 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:35.602888 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:35.603048 1859 server_base.cc:1034] running on GCE node
W20260618 23:31:35.602876 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:35.603404 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:35.603715 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:35.604918 1859 hybrid_clock.cc:648] HybridClock initialized: now 1781825495604908 us; error 63 us; skew 500 ppm
Jun 18 23:31:35 dist-test-slave-6224 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:35.607690 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:31:35.608198 1859 webserver.cc:460] Webserver started at http://127.0.98.3:37149/ using document root <none> and password file <none>
I20260618 23:31:35.608469 1859 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:35.608520 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:35.608654 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:35.609771 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "51c9ab34b4cf46ba852e7d848e0a11c4"
format_stamp: "Formatted at 2026-06-18 23:31:35 on dist-test-slave-6224"
server_key: "d952d73b911980c5db195ae920e5211a"
server_key_iv: "f8be39f1cb939dbf82fe4aa1c2e20e7a"
server_key_version: "encryptionkey@0"
I20260618 23:31:35.610136 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "51c9ab34b4cf46ba852e7d848e0a11c4"
format_stamp: "Formatted at 2026-06-18 23:31:35 on dist-test-slave-6224"
server_key: "d952d73b911980c5db195ae920e5211a"
server_key_iv: "f8be39f1cb939dbf82fe4aa1c2e20e7a"
server_key_version: "encryptionkey@0"
I20260618 23:31:35.611697 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260618 23:31:35.612726 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:35.612977 1859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:35.613112 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "51c9ab34b4cf46ba852e7d848e0a11c4"
format_stamp: "Formatted at 2026-06-18 23:31:35 on dist-test-slave-6224"
server_key: "d952d73b911980c5db195ae920e5211a"
server_key_iv: "f8be39f1cb939dbf82fe4aa1c2e20e7a"
server_key_version: "encryptionkey@0"
I20260618 23:31:35.613206 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:35.635208 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:35.637845 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:35.637985 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:35.638569 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:35.638612 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:35.638638 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:35.638650 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 23:31:35 dist-test-slave-6224 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:35.647470 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32927
I20260618 23:31:35.647941 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32927 every 8 connection(s)
I20260618 23:31:35.648033 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:31:35.650581 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.639263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39225 (local address 127.0.98.62:33725)
0618 23:31:35.639362 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:35.639365 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:35.639532 (+ 167us) server_negotiation.cc:407] Connection header received
0618 23:31:35.640206 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:35.640208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:35.640223 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:35.640342 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:35.641382 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.642280 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.643967 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.644189 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.644270 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.647228 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:35.647231 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:35.647232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:35.647240 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:35.649030 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:35.649522 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:35.649523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:35.649524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:35.649590 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:35.649850 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:35.649851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:35.649852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:35.649993 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:35.650080 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:35.650382 (+ 302us) server_negotiation.cc:299] Negotiation successful
0618 23:31:35.650476 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 23:31:35.650816 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.639168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33725 (local address 127.0.98.3:39225)
0618 23:31:35.639372 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:35.639426 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:35.640052 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:35.640429 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:35.640432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:35.640919 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:35.641244 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.641250 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.642426 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.642427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.643852 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.643855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.644299 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.644300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.644470 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.645203 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:35.645212 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:35.647059 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:35.649147 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:35.649153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:35.649156 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:35.649423 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:35.649658 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:35.649659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:35.649659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:35.649780 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:35.650063 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:35.650074 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:35.650239 (+ 165us) client_negotiation.cc:769] Sending connection context
0618 23:31:35.650358 (+ 119us) client_negotiation.cc:240] Negotiation successful
0618 23:31:35.650544 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":139,"threads_started":1}
I20260618 23:31:35.651345 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33725
I20260618 23:31:35.651463 1990 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:35.651719 1990 heartbeater.cc:511] Master 127.0.98.62:33725 requested a full tablet report, sending...
I20260618 23:31:35.652416 1506 ts_manager.cc:194] Registered new tserver with Master: 51c9ab34b4cf46ba852e7d848e0a11c4 (127.0.98.3:32927)
I20260618 23:31:35.653023 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39225
I20260618 23:31:35.656243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 1859
I20260618 23:31:35.656348 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:31:35.656517 392 external_mini_cluster.cc:1428] Setting key f378fd11bb33aaeff13370c30acf0b30
I20260618 23:31:35.657608 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:31:35.659013 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.652605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33725 (local address 127.0.98.3:57765)
0618 23:31:35.652795 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:35.652811 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:35.652876 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:35.653023 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:35.653024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:35.653261 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:35.653341 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.653344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.654067 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.654068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.654558 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.654560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.655081 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.655082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.655127 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.655611 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:35.655616 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:35.655900 (+ 284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:35.657862 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:35.657863 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:35.657864 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:35.658067 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:35.658304 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:35.658305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:35.658305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:35.658354 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:35.658651 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:35.658653 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:35.658715 (+ 62us) client_negotiation.cc:769] Sending connection context
0618 23:31:35.658804 (+ 89us) client_negotiation.cc:240] Negotiation successful
0618 23:31:35.658922 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":96,"threads_started":1}
I20260618 23:31:35.659082 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.652828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57765 (local address 127.0.98.62:33725)
0618 23:31:35.652889 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:35.652891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:35.652900 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:31:35.652929 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:35.652930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:35.652940 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:35.653016 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:35.653444 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.653991 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.654680 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.654989 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.655067 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.655988 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:35.655989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:35.655990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:35.655996 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:35.657782 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:35.658148 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:35.658149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:35.658149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:35.658220 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:35.658401 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:35.658401 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:35.658402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:35.658552 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:35.658632 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:35.658844 (+ 212us) server_negotiation.cc:299] Negotiation successful
0618 23:31:35.658939 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Jun 18 23:31:35 dist-test-slave-6224 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825495, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 23:31:35 dist-test-slave-6224 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825495, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:35.694638 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.685549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45678 (local address 127.0.98.62:33725)
0618 23:31:35.685639 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:35.685642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:35.685657 (+ 15us) server_negotiation.cc:407] Connection header received
0618 23:31:35.685697 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:35.685699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:35.685716 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:35.685912 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:35.686841 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.687451 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.688347 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.688738 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.688859 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.691635 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:35.691637 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:35.691638 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:35.691651 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:35.693218 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:35.693607 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:35.693608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:35.693609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:35.693675 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:35.693991 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:35.693992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:35.693992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:35.694142 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:35.694236 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:35.694424 (+ 188us) server_negotiation.cc:299] Negotiation successful
0618 23:31:35.694505 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 23:31:35.697420 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:31:35.700425 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:31:35.709997 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.704786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32927 (local address 127.0.0.1:47158)
0618 23:31:35.705397 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:35.705420 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:35.705465 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:35.705711 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:35.705715 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:35.705717 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:35.705800 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.705802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.707047 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.707049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.708520 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.708523 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.709573 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.709574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.709689 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.709709 (+ 20us) client_negotiation.cc:769] Sending connection context
0618 23:31:35.709797 (+ 88us) client_negotiation.cc:240] Negotiation successful
0618 23:31:35.709880 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":4,"thread_start_us":111,"threads_started":1}
I20260618 23:31:35.710140 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.704787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45933 (local address 127.0.0.1:39274)
0618 23:31:35.705145 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:35.705190 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:35.705271 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:35.705946 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:35.705947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:35.705951 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:35.706040 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.706042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.707476 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.707477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.708709 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.708711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.709759 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.709761 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.709843 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.709849 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:31:35.709923 (+ 74us) client_negotiation.cc:240] Negotiation successful
0618 23:31:35.709986 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":82,"threads_started":1}
I20260618 23:31:35.710243 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.704932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39274 (local address 127.0.98.1:45933)
0618 23:31:35.705709 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:35.705717 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:35.705731 (+ 14us) server_negotiation.cc:407] Connection header received
0618 23:31:35.705785 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:35.705788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:35.705807 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:35.706123 (+ 316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:35.706200 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.707359 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.708824 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.709675 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.709744 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.709816 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:35.709939 (+ 123us) server_negotiation.cc:299] Negotiation successful
0618 23:31:35.709980 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":723,"thread_start_us":96,"threads_started":1}
I20260618 23:31:35.710398 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.705115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47158 (local address 127.0.98.3:32927)
0618 23:31:35.705527 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:35.705533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:35.705544 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:31:35.705591 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:35.705594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:35.705612 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:35.705701 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:35.705876 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.706974 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.708656 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.709468 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.709564 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.709657 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:35.710194 (+ 537us) server_negotiation.cc:299] Negotiation successful
0618 23:31:35.710285 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":115,"threads_started":1}
I20260618 23:31:35.710915 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.705071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38663 (local address 127.0.0.1:55828)
0618 23:31:35.706503 (+ 1432us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:35.706519 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:35.706588 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:35.706874 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:35.706875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:35.706877 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:35.706925 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.706926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.708110 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.708111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.709043 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.709045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.710025 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.710026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.710716 (+ 690us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.710726 (+ 10us) client_negotiation.cc:769] Sending connection context
0618 23:31:35.710794 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 23:31:35.710848 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1410,"thread_start_us":69,"threads_started":1}
I20260618 23:31:35.711071 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5dc807b5d940da991fcb24d778fb73 (DEFAULT_TABLE table=test-table [id=07576b3f389247d8a5c1394744d1dfb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:35.711140 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5dc807b5d940da991fcb24d778fb73 (DEFAULT_TABLE table=test-table [id=07576b3f389247d8a5c1394744d1dfb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:35.711596 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5dc807b5d940da991fcb24d778fb73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:35.711653 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5dc807b5d940da991fcb24d778fb73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:35.711773 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.705433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55828 (local address 127.0.98.2:38663)
0618 23:31:35.705656 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:35.705661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:35.706649 (+ 988us) server_negotiation.cc:407] Connection header received
0618 23:31:35.706702 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:35.706704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:35.706721 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:35.706805 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:35.706997 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.708015 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.709146 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.709842 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.709906 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.709986 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:35.711349 (+ 1363us) server_negotiation.cc:299] Negotiation successful
0618 23:31:35.711506 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":114,"threads_started":1}
I20260618 23:31:35.715075 2013 tablet_bootstrap.cc:492] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4: Bootstrap starting.
I20260618 23:31:35.715502 2014 tablet_bootstrap.cc:492] T 8e5dc807b5d940da991fcb24d778fb73 P 7f1309bc848a44b1808e1e740c132d27: Bootstrap starting.
I20260618 23:31:35.715703 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 8e5dc807b5d940da991fcb24d778fb73 (DEFAULT_TABLE table=test-table [id=07576b3f389247d8a5c1394744d1dfb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:35.715899 2013 tablet_bootstrap.cc:654] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:35.716161 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e5dc807b5d940da991fcb24d778fb73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:35.716351 2014 tablet_bootstrap.cc:654] T 8e5dc807b5d940da991fcb24d778fb73 P 7f1309bc848a44b1808e1e740c132d27: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:35.716342 2013 log.cc:824] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:35.716738 2014 log.cc:824] T 8e5dc807b5d940da991fcb24d778fb73 P 7f1309bc848a44b1808e1e740c132d27: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:35.717665 2014 tablet_bootstrap.cc:492] T 8e5dc807b5d940da991fcb24d778fb73 P 7f1309bc848a44b1808e1e740c132d27: No bootstrap required, opened a new log
I20260618 23:31:35.717738 2014 ts_tablet_manager.cc:1397] T 8e5dc807b5d940da991fcb24d778fb73 P 7f1309bc848a44b1808e1e740c132d27: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:31:35.719431 2014 raft_consensus.cc:348] T 8e5dc807b5d940da991fcb24d778fb73 P 7f1309bc848a44b1808e1e740c132d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } } peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } }
I20260618 23:31:35.719655 2014 raft_consensus.cc:374] T 8e5dc807b5d940da991fcb24d778fb73 P 7f1309bc848a44b1808e1e740c132d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:35.719700 2014 raft_consensus.cc:729] T 8e5dc807b5d940da991fcb24d778fb73 P 7f1309bc848a44b1808e1e740c132d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f1309bc848a44b1808e1e740c132d27, State: Initialized, Role: FOLLOWER
I20260618 23:31:35.719882 2014 consensus_queue.cc:260] T 8e5dc807b5d940da991fcb24d778fb73 P 7f1309bc848a44b1808e1e740c132d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } } peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } }
I20260618 23:31:35.720225 2014 ts_tablet_manager.cc:1428] T 8e5dc807b5d940da991fcb24d778fb73 P 7f1309bc848a44b1808e1e740c132d27: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260618 23:31:35.720295 1710 heartbeater.cc:503] Master 127.0.98.62:33725 was elected leader, sending a full tablet report...
I20260618 23:31:35.720628 2013 tablet_bootstrap.cc:492] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4: No bootstrap required, opened a new log
I20260618 23:31:35.720713 2013 ts_tablet_manager.cc:1397] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260618 23:31:35.722690 2013 raft_consensus.cc:348] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } } peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } }
I20260618 23:31:35.722885 2013 raft_consensus.cc:374] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:35.722942 2013 raft_consensus.cc:729] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c9ab34b4cf46ba852e7d848e0a11c4, State: Initialized, Role: FOLLOWER
I20260618 23:31:35.723088 2013 consensus_queue.cc:260] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } } peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } }
I20260618 23:31:35.723392 2013 ts_tablet_manager.cc:1428] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:31:35.723495 1990 heartbeater.cc:503] Master 127.0.98.62:33725 was elected leader, sending a full tablet report...
I20260618 23:31:35.723661 2017 tablet_bootstrap.cc:492] T 8e5dc807b5d940da991fcb24d778fb73 P 233d75d48e4b465381298a1f2e2ff5d2: Bootstrap starting.
I20260618 23:31:35.724457 2017 tablet_bootstrap.cc:654] T 8e5dc807b5d940da991fcb24d778fb73 P 233d75d48e4b465381298a1f2e2ff5d2: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:35.724825 2017 log.cc:824] T 8e5dc807b5d940da991fcb24d778fb73 P 233d75d48e4b465381298a1f2e2ff5d2: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:35.725762 2017 tablet_bootstrap.cc:492] T 8e5dc807b5d940da991fcb24d778fb73 P 233d75d48e4b465381298a1f2e2ff5d2: No bootstrap required, opened a new log
I20260618 23:31:35.725824 2017 ts_tablet_manager.cc:1397] T 8e5dc807b5d940da991fcb24d778fb73 P 233d75d48e4b465381298a1f2e2ff5d2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:31:35.727321 2017 raft_consensus.cc:348] T 8e5dc807b5d940da991fcb24d778fb73 P 233d75d48e4b465381298a1f2e2ff5d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } } peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } }
I20260618 23:31:35.727475 2017 raft_consensus.cc:374] T 8e5dc807b5d940da991fcb24d778fb73 P 233d75d48e4b465381298a1f2e2ff5d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:35.727501 2017 raft_consensus.cc:729] T 8e5dc807b5d940da991fcb24d778fb73 P 233d75d48e4b465381298a1f2e2ff5d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 233d75d48e4b465381298a1f2e2ff5d2, State: Initialized, Role: FOLLOWER
I20260618 23:31:35.727682 2017 consensus_queue.cc:260] T 8e5dc807b5d940da991fcb24d778fb73 P 233d75d48e4b465381298a1f2e2ff5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } } peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } }
I20260618 23:31:35.728046 2017 ts_tablet_manager.cc:1428] T 8e5dc807b5d940da991fcb24d778fb73 P 233d75d48e4b465381298a1f2e2ff5d2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:31:35.728132 1850 heartbeater.cc:503] Master 127.0.98.62:33725 was elected leader, sending a full tablet report...
I20260618 23:31:35.738514 2019 raft_consensus.cc:482] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:31:35.738689 2019 raft_consensus.cc:504] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } } peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } }
I20260618 23:31:35.739151 2019 leader_election.cc:288] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f1309bc848a44b1808e1e740c132d27 (127.0.98.1:45933), 233d75d48e4b465381298a1f2e2ff5d2 (127.0.98.2:38663)
I20260618 23:31:35.742278 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.739481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53347 (local address 127.0.98.1:45933)
0618 23:31:35.739595 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:35.739598 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:35.739617 (+ 19us) server_negotiation.cc:407] Connection header received
0618 23:31:35.739674 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:35.739675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:35.739688 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:35.739756 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:35.739949 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.740444 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.741528 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.741906 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.741996 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.742006 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:35.742161 (+ 155us) server_negotiation.cc:299] Negotiation successful
0618 23:31:35.742199 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260618 23:31:35.742291 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.739390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45933 (local address 127.0.98.3:53347)
0618 23:31:35.739453 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:35.739476 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:35.739596 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:35.739778 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:35.739780 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:35.739783 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:35.739866 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.739869 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.740542 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.740543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.741419 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.741422 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.741994 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.741995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.742075 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.742078 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:31:35.742154 (+ 76us) client_negotiation.cc:240] Negotiation successful
0618 23:31:35.742201 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260618 23:31:35.742512 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.739689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51093 (local address 127.0.98.2:38663)
0618 23:31:35.739764 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:35.739766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:35.739872 (+ 106us) server_negotiation.cc:407] Connection header received
0618 23:31:35.739952 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:35.739953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:35.739965 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:35.740054 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:35.740234 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.740739 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.741728 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.742109 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.742194 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.742207 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:35.742368 (+ 161us) server_negotiation.cc:299] Negotiation successful
0618 23:31:35.742429 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 23:31:35.742527 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.739539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38663 (local address 127.0.98.3:51093)
0618 23:31:35.739786 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:35.739797 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:35.739884 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:35.740054 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:35.740055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:35.740057 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:35.740155 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.740157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.740823 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.740824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.741628 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.741630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.742194 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.742195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.742276 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.742280 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:31:35.742356 (+ 76us) client_negotiation.cc:240] Negotiation successful
0618 23:31:35.742404 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":32,"thread_start_us":135,"threads_started":1}
I20260618 23:31:35.742664 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5dc807b5d940da991fcb24d778fb73" candidate_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f1309bc848a44b1808e1e740c132d27" is_pre_election: true
I20260618 23:31:35.742854 1664 raft_consensus.cc:2454] T 8e5dc807b5d940da991fcb24d778fb73 P 7f1309bc848a44b1808e1e740c132d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51c9ab34b4cf46ba852e7d848e0a11c4 in term 0.
I20260618 23:31:35.742893 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5dc807b5d940da991fcb24d778fb73" candidate_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "233d75d48e4b465381298a1f2e2ff5d2" is_pre_election: true
I20260618 23:31:35.743093 1804 raft_consensus.cc:2454] T 8e5dc807b5d940da991fcb24d778fb73 P 233d75d48e4b465381298a1f2e2ff5d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51c9ab34b4cf46ba852e7d848e0a11c4 in term 0.
I20260618 23:31:35.743162 1876 leader_election.cc:302] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51c9ab34b4cf46ba852e7d848e0a11c4, 7f1309bc848a44b1808e1e740c132d27; no voters:
I20260618 23:31:35.743323 2019 raft_consensus.cc:2781] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:31:35.743450 2019 raft_consensus.cc:482] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:31:35.743495 2019 raft_consensus.cc:3037] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:35.744491 2019 raft_consensus.cc:504] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } } peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } }
I20260618 23:31:35.744678 2019 leader_election.cc:288] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4 [CANDIDATE]: Term 1 election: Requested vote from peers 7f1309bc848a44b1808e1e740c132d27 (127.0.98.1:45933), 233d75d48e4b465381298a1f2e2ff5d2 (127.0.98.2:38663)
I20260618 23:31:35.744969 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5dc807b5d940da991fcb24d778fb73" candidate_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f1309bc848a44b1808e1e740c132d27"
I20260618 23:31:35.745096 1664 raft_consensus.cc:3037] T 8e5dc807b5d940da991fcb24d778fb73 P 7f1309bc848a44b1808e1e740c132d27 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:35.746021 1664 raft_consensus.cc:2454] T 8e5dc807b5d940da991fcb24d778fb73 P 7f1309bc848a44b1808e1e740c132d27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51c9ab34b4cf46ba852e7d848e0a11c4 in term 1.
I20260618 23:31:35.746181 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e5dc807b5d940da991fcb24d778fb73" candidate_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "233d75d48e4b465381298a1f2e2ff5d2"
I20260618 23:31:35.746217 1876 leader_election.cc:302] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51c9ab34b4cf46ba852e7d848e0a11c4, 7f1309bc848a44b1808e1e740c132d27; no voters:
I20260618 23:31:35.746287 1804 raft_consensus.cc:3037] T 8e5dc807b5d940da991fcb24d778fb73 P 233d75d48e4b465381298a1f2e2ff5d2 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:35.746343 2019 raft_consensus.cc:2781] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:35.746519 2019 raft_consensus.cc:686] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 1 LEADER]: Becoming Leader. State: Replica: 51c9ab34b4cf46ba852e7d848e0a11c4, State: Running, Role: LEADER
I20260618 23:31:35.746631 2019 consensus_queue.cc:237] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } } peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } }
I20260618 23:31:35.747316 1804 raft_consensus.cc:2454] T 8e5dc807b5d940da991fcb24d778fb73 P 233d75d48e4b465381298a1f2e2ff5d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51c9ab34b4cf46ba852e7d848e0a11c4 in term 1.
I20260618 23:31:35.747761 1506 catalog_manager.cc:5495] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51c9ab34b4cf46ba852e7d848e0a11c4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } health_report { overall_health: UNKNOWN } } }
I20260618 23:31:35.759043 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.756265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47172 (local address 127.0.98.3:32927)
0618 23:31:35.756346 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:35.756348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:35.756399 (+ 51us) server_negotiation.cc:407] Connection header received
0618 23:31:35.756511 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:35.756513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:35.756536 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:35.756609 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:31:35.756812 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.757338 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.758181 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.758383 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.758456 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.758636 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:31:35.758759 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:31:35.758832 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:35.758883 (+ 51us) server_negotiation.cc:299] Negotiation successful
0618 23:31:35.758906 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 23:31:35.761433 1664 raft_consensus.cc:1261] T 8e5dc807b5d940da991fcb24d778fb73 P 7f1309bc848a44b1808e1e740c132d27 [term 1 FOLLOWER]: Refusing update from remote peer 51c9ab34b4cf46ba852e7d848e0a11c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:31:35.761504 1804 raft_consensus.cc:1261] T 8e5dc807b5d940da991fcb24d778fb73 P 233d75d48e4b465381298a1f2e2ff5d2 [term 1 FOLLOWER]: Refusing update from remote peer 51c9ab34b4cf46ba852e7d848e0a11c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:31:35.761861 2023 consensus_queue.cc:1035] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:31:35.762044 2019 consensus_queue.cc:1035] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:31:35.765540 2027 mvcc.cc:205] Tried to move back new op lower bound from 7298357230635372544 to 7298357230579060736. Current Snapshot: MvccSnapshot[applied={T|T < 7298357230635372544}]
I20260618 23:31:35.765514 2028 mvcc.cc:205] Tried to move back new op lower bound from 7298357230635372544 to 7298357230579060736. Current Snapshot: MvccSnapshot[applied={T|T < 7298357230635372544}]
I20260618 23:31:35.770123 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:31:35.770247 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:31:35.771129 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771: Sending DeleteTablet for 3 replicas of tablet 8e5dc807b5d940da991fcb24d778fb73
I20260618 23:31:35.771539 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e5dc807b5d940da991fcb24d778fb73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55828
I20260618 23:31:35.771538 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e5dc807b5d940da991fcb24d778fb73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39274
I20260618 23:31:35.771513 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e5dc807b5d940da991fcb24d778fb73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47158
I20260618 23:31:35.771931 2037 tablet_replica.cc:330] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4: stopping tablet replica
I20260618 23:31:35.772068 2037 raft_consensus.cc:2229] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:31:35.772220 2037 raft_consensus.cc:2258] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:35.772193 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:31:35.772357 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:31:35.772629 2038 tablet_replica.cc:330] T 8e5dc807b5d940da991fcb24d778fb73 P 233d75d48e4b465381298a1f2e2ff5d2: stopping tablet replica
I20260618 23:31:35.772761 2038 raft_consensus.cc:2229] T 8e5dc807b5d940da991fcb24d778fb73 P 233d75d48e4b465381298a1f2e2ff5d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:31:35.772889 2038 raft_consensus.cc:2258] T 8e5dc807b5d940da991fcb24d778fb73 P 233d75d48e4b465381298a1f2e2ff5d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:35.773137 2036 tablet_replica.cc:330] T 8e5dc807b5d940da991fcb24d778fb73 P 7f1309bc848a44b1808e1e740c132d27: stopping tablet replica
I20260618 23:31:35.773248 2036 raft_consensus.cc:2229] T 8e5dc807b5d940da991fcb24d778fb73 P 7f1309bc848a44b1808e1e740c132d27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:31:35.773454 2038 ts_tablet_manager.cc:1905] T 8e5dc807b5d940da991fcb24d778fb73 P 233d75d48e4b465381298a1f2e2ff5d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:31:35.773525 2037 ts_tablet_manager.cc:1905] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:31:35.774225 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 2cee7c153e55483097f52cc5e2cce339 (DEFAULT_TABLE table=test-table [id=c7bab42c4fe344a7baf297a86f866378]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:35.774241 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 2cee7c153e55483097f52cc5e2cce339 (DEFAULT_TABLE table=test-table [id=c7bab42c4fe344a7baf297a86f866378]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:35.774389 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cee7c153e55483097f52cc5e2cce339. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:35.774420 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cee7c153e55483097f52cc5e2cce339. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:35.775538 2037 ts_tablet_manager.cc:1918] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:31:35.775645 2037 log.cc:1196] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/8e5dc807b5d940da991fcb24d778fb73
I20260618 23:31:35.775808 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 2cee7c153e55483097f52cc5e2cce339 (DEFAULT_TABLE table=test-table [id=c7bab42c4fe344a7baf297a86f866378]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:35.775928 2037 ts_tablet_manager.cc:1939] T 8e5dc807b5d940da991fcb24d778fb73 P 51c9ab34b4cf46ba852e7d848e0a11c4: Deleting consensus metadata
I20260618 23:31:35.775946 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cee7c153e55483097f52cc5e2cce339. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:35.775930 2038 ts_tablet_manager.cc:1918] T 8e5dc807b5d940da991fcb24d778fb73 P 233d75d48e4b465381298a1f2e2ff5d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:31:35.776006 2038 log.cc:1196] T 8e5dc807b5d940da991fcb24d778fb73 P 233d75d48e4b465381298a1f2e2ff5d2: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/8e5dc807b5d940da991fcb24d778fb73
I20260618 23:31:35.776389 2017 tablet_bootstrap.cc:492] T 2cee7c153e55483097f52cc5e2cce339 P 233d75d48e4b465381298a1f2e2ff5d2: Bootstrap starting.
I20260618 23:31:35.776471 1491 catalog_manager.cc:4841] TS 51c9ab34b4cf46ba852e7d848e0a11c4 (127.0.98.3:32927): tablet 8e5dc807b5d940da991fcb24d778fb73 (table test-table [id=07576b3f389247d8a5c1394744d1dfb6]) successfully deleted
I20260618 23:31:35.776969 2017 tablet_bootstrap.cc:654] T 2cee7c153e55483097f52cc5e2cce339 P 233d75d48e4b465381298a1f2e2ff5d2: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:35.777423 2038 ts_tablet_manager.cc:1939] T 8e5dc807b5d940da991fcb24d778fb73 P 233d75d48e4b465381298a1f2e2ff5d2: Deleting consensus metadata
I20260618 23:31:35.778033 2013 tablet_bootstrap.cc:492] T 2cee7c153e55483097f52cc5e2cce339 P 51c9ab34b4cf46ba852e7d848e0a11c4: Bootstrap starting.
I20260618 23:31:35.778208 1491 catalog_manager.cc:4841] TS 233d75d48e4b465381298a1f2e2ff5d2 (127.0.98.2:38663): tablet 8e5dc807b5d940da991fcb24d778fb73 (table test-table [id=07576b3f389247d8a5c1394744d1dfb6]) successfully deleted
I20260618 23:31:35.778481 2013 tablet_bootstrap.cc:654] T 2cee7c153e55483097f52cc5e2cce339 P 51c9ab34b4cf46ba852e7d848e0a11c4: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:35.779170 2014 tablet_bootstrap.cc:492] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27: Bootstrap starting.
I20260618 23:31:35.779633 2014 tablet_bootstrap.cc:654] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:35.779784 2036 raft_consensus.cc:2258] T 8e5dc807b5d940da991fcb24d778fb73 P 7f1309bc848a44b1808e1e740c132d27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:35.780378 2013 tablet_bootstrap.cc:492] T 2cee7c153e55483097f52cc5e2cce339 P 51c9ab34b4cf46ba852e7d848e0a11c4: No bootstrap required, opened a new log
I20260618 23:31:35.780442 2013 ts_tablet_manager.cc:1397] T 2cee7c153e55483097f52cc5e2cce339 P 51c9ab34b4cf46ba852e7d848e0a11c4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:31:35.780453 2014 tablet_bootstrap.cc:492] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27: No bootstrap required, opened a new log
I20260618 23:31:35.780490 2014 ts_tablet_manager.cc:1397] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260618 23:31:35.780612 2013 raft_consensus.cc:348] T 2cee7c153e55483097f52cc5e2cce339 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } }
I20260618 23:31:35.780653 2014 raft_consensus.cc:348] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } }
I20260618 23:31:35.780702 2013 raft_consensus.cc:374] T 2cee7c153e55483097f52cc5e2cce339 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:35.780719 2014 raft_consensus.cc:374] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:35.780727 2013 raft_consensus.cc:729] T 2cee7c153e55483097f52cc5e2cce339 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c9ab34b4cf46ba852e7d848e0a11c4, State: Initialized, Role: FOLLOWER
I20260618 23:31:35.780745 2014 raft_consensus.cc:729] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f1309bc848a44b1808e1e740c132d27, State: Initialized, Role: FOLLOWER
I20260618 23:31:35.780782 2013 consensus_queue.cc:260] T 2cee7c153e55483097f52cc5e2cce339 P 51c9ab34b4cf46ba852e7d848e0a11c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } }
I20260618 23:31:35.780807 2014 consensus_queue.cc:260] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } }
I20260618 23:31:35.780879 2013 ts_tablet_manager.cc:1428] T 2cee7c153e55483097f52cc5e2cce339 P 51c9ab34b4cf46ba852e7d848e0a11c4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260618 23:31:35.780903 2014 ts_tablet_manager.cc:1428] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:35.781045 2036 ts_tablet_manager.cc:1905] T 8e5dc807b5d940da991fcb24d778fb73 P 7f1309bc848a44b1808e1e740c132d27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:31:35.781404 2017 tablet_bootstrap.cc:492] T 2cee7c153e55483097f52cc5e2cce339 P 233d75d48e4b465381298a1f2e2ff5d2: No bootstrap required, opened a new log
I20260618 23:31:35.781447 2017 ts_tablet_manager.cc:1397] T 2cee7c153e55483097f52cc5e2cce339 P 233d75d48e4b465381298a1f2e2ff5d2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260618 23:31:35.781595 2017 raft_consensus.cc:348] T 2cee7c153e55483097f52cc5e2cce339 P 233d75d48e4b465381298a1f2e2ff5d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } }
I20260618 23:31:35.781665 2017 raft_consensus.cc:374] T 2cee7c153e55483097f52cc5e2cce339 P 233d75d48e4b465381298a1f2e2ff5d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:35.781685 2017 raft_consensus.cc:729] T 2cee7c153e55483097f52cc5e2cce339 P 233d75d48e4b465381298a1f2e2ff5d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 233d75d48e4b465381298a1f2e2ff5d2, State: Initialized, Role: FOLLOWER
I20260618 23:31:35.781735 2017 consensus_queue.cc:260] T 2cee7c153e55483097f52cc5e2cce339 P 233d75d48e4b465381298a1f2e2ff5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } }
I20260618 23:31:35.781826 2017 ts_tablet_manager.cc:1428] T 2cee7c153e55483097f52cc5e2cce339 P 233d75d48e4b465381298a1f2e2ff5d2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260618 23:31:35.782676 2036 ts_tablet_manager.cc:1918] T 8e5dc807b5d940da991fcb24d778fb73 P 7f1309bc848a44b1808e1e740c132d27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:31:35.782742 2036 log.cc:1196] T 8e5dc807b5d940da991fcb24d778fb73 P 7f1309bc848a44b1808e1e740c132d27: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-0/wal/wals/8e5dc807b5d940da991fcb24d778fb73
I20260618 23:31:35.783016 2036 ts_tablet_manager.cc:1939] T 8e5dc807b5d940da991fcb24d778fb73 P 7f1309bc848a44b1808e1e740c132d27: Deleting consensus metadata
I20260618 23:31:35.783463 1492 catalog_manager.cc:4841] TS 7f1309bc848a44b1808e1e740c132d27 (127.0.98.1:45933): tablet 8e5dc807b5d940da991fcb24d778fb73 (table test-table [id=07576b3f389247d8a5c1394744d1dfb6]) successfully deleted
W20260618 23:31:35.813709 1711 tablet.cc:2369] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:31:35.898592 1991 tablet.cc:2369] T 2cee7c153e55483097f52cc5e2cce339 P 51c9ab34b4cf46ba852e7d848e0a11c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:31:35.924057 2018 raft_consensus.cc:482] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:31:35.924252 2018 raft_consensus.cc:504] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } }
I20260618 23:31:35.924789 2018 leader_election.cc:288] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 233d75d48e4b465381298a1f2e2ff5d2 (127.0.98.2:38663), 51c9ab34b4cf46ba852e7d848e0a11c4 (127.0.98.3:32927)
I20260618 23:31:35.928179 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.925484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51861 (local address 127.0.98.3:32927)
0618 23:31:35.925544 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:35.925546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:35.925561 (+ 15us) server_negotiation.cc:407] Connection header received
0618 23:31:35.925586 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:35.925587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:35.925594 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:35.925661 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:35.925841 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.926332 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.927375 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.927816 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.927896 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.927905 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:35.928066 (+ 161us) server_negotiation.cc:299] Negotiation successful
0618 23:31:35.928103 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260618 23:31:35.928192 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.925269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32927 (local address 127.0.98.1:51861)
0618 23:31:35.925403 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:35.925414 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:35.925457 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:35.925670 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:35.925671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:35.925674 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:35.925759 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.925761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.926424 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.926426 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.927280 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.927282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.927898 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.927899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.927982 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.927993 (+ 11us) client_negotiation.cc:769] Sending connection context
0618 23:31:35.928061 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 23:31:35.928109 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
I20260618 23:31:35.928514 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.925271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47903 (local address 127.0.98.2:38663)
0618 23:31:35.925349 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:35.925352 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:35.925364 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:31:35.925402 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:35.925404 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:35.925416 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:35.925518 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:35.925726 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.926257 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.927657 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.928043 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.928135 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.928150 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:35.928363 (+ 213us) server_negotiation.cc:299] Negotiation successful
0618 23:31:35.928426 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 23:31:35.928690 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cee7c153e55483097f52cc5e2cce339" candidate_uuid: "7f1309bc848a44b1808e1e740c132d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" is_pre_election: true
I20260618 23:31:35.928808 2042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.925062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38663 (local address 127.0.98.1:47903)
0618 23:31:35.925200 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:35.925214 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:35.925271 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:35.925528 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:35.925529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:35.925533 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:35.925638 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.925640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.926352 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.926353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.927475 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.927478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.928150 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.928152 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.928238 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.928242 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:31:35.928652 (+ 410us) client_negotiation.cc:240] Negotiation successful
0618 23:31:35.928714 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":15,"thread_start_us":87,"threads_started":1}
I20260618 23:31:35.928890 1944 raft_consensus.cc:2454] T 2cee7c153e55483097f52cc5e2cce339 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f1309bc848a44b1808e1e740c132d27 in term 0.
I20260618 23:31:35.929108 1595 leader_election.cc:302] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51c9ab34b4cf46ba852e7d848e0a11c4, 7f1309bc848a44b1808e1e740c132d27; no voters:
I20260618 23:31:35.929126 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cee7c153e55483097f52cc5e2cce339" candidate_uuid: "7f1309bc848a44b1808e1e740c132d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "233d75d48e4b465381298a1f2e2ff5d2" is_pre_election: true
I20260618 23:31:35.929209 1804 raft_consensus.cc:2454] T 2cee7c153e55483097f52cc5e2cce339 P 233d75d48e4b465381298a1f2e2ff5d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f1309bc848a44b1808e1e740c132d27 in term 0.
I20260618 23:31:35.929263 2018 raft_consensus.cc:2781] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:31:35.929304 2018 raft_consensus.cc:482] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:31:35.929327 2018 raft_consensus.cc:3037] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:35.930163 2018 raft_consensus.cc:504] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } }
I20260618 23:31:35.930326 2018 leader_election.cc:288] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27 [CANDIDATE]: Term 1 election: Requested vote from peers 233d75d48e4b465381298a1f2e2ff5d2 (127.0.98.2:38663), 51c9ab34b4cf46ba852e7d848e0a11c4 (127.0.98.3:32927)
I20260618 23:31:35.930526 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cee7c153e55483097f52cc5e2cce339" candidate_uuid: "7f1309bc848a44b1808e1e740c132d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "233d75d48e4b465381298a1f2e2ff5d2"
I20260618 23:31:35.930559 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cee7c153e55483097f52cc5e2cce339" candidate_uuid: "7f1309bc848a44b1808e1e740c132d27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4"
I20260618 23:31:35.930619 1804 raft_consensus.cc:3037] T 2cee7c153e55483097f52cc5e2cce339 P 233d75d48e4b465381298a1f2e2ff5d2 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:35.930641 1944 raft_consensus.cc:3037] T 2cee7c153e55483097f52cc5e2cce339 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:35.931507 1804 raft_consensus.cc:2454] T 2cee7c153e55483097f52cc5e2cce339 P 233d75d48e4b465381298a1f2e2ff5d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f1309bc848a44b1808e1e740c132d27 in term 1.
I20260618 23:31:35.931771 1944 raft_consensus.cc:2454] T 2cee7c153e55483097f52cc5e2cce339 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f1309bc848a44b1808e1e740c132d27 in term 1.
I20260618 23:31:35.931881 1595 leader_election.cc:302] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 233d75d48e4b465381298a1f2e2ff5d2, 7f1309bc848a44b1808e1e740c132d27; no voters:
I20260618 23:31:35.932000 2018 raft_consensus.cc:2781] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:35.932171 2018 raft_consensus.cc:686] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27 [term 1 LEADER]: Becoming Leader. State: Replica: 7f1309bc848a44b1808e1e740c132d27, State: Running, Role: LEADER
I20260618 23:31:35.932267 2018 consensus_queue.cc:237] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } }
I20260618 23:31:35.933154 1506 catalog_manager.cc:5495] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f1309bc848a44b1808e1e740c132d27 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7f1309bc848a44b1808e1e740c132d27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } health_report { overall_health: UNKNOWN } } }
W20260618 23:31:35.975406 1851 tablet.cc:2369] T 2cee7c153e55483097f52cc5e2cce339 P 233d75d48e4b465381298a1f2e2ff5d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 18 23:31:35 dist-test-slave-6224 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825495, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:35.999447 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.988598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33725 (local address 127.0.0.1:45686)
0618 23:31:35.988808 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:35.988821 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:35.988934 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:35.989144 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:35.989145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:35.989471 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:35.989528 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.989530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.990272 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.990273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.990818 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:35.990819 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.991183 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.991184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:35.991226 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.994562 (+ 3336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:35.994567 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:35.996216 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:35.998293 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:35.998320 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:35.998322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:35.998544 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:35.998774 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:35.998775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:35.998775 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:35.998844 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:35.999121 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:35.999127 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:35.999181 (+ 54us) client_negotiation.cc:769] Sending connection context
0618 23:31:35.999244 (+ 63us) client_negotiation.cc:240] Negotiation successful
0618 23:31:35.999367 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":131,"threads_started":1}
I20260618 23:31:35.999591 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:35.988705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45686 (local address 127.0.98.62:33725)
0618 23:31:35.988794 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:35.988797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:35.988961 (+ 164us) server_negotiation.cc:407] Connection header received
0618 23:31:35.989012 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:35.989013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:35.989026 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:35.989121 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:35.989623 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.990187 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.990921 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:35.991105 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:35.991175 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:35.996352 (+ 5177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:35.996354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:35.996354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:35.996362 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:35.998186 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:35.998637 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:35.998638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:35.998638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:35.998700 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:35.998909 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:35.998910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:35.998910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:35.999059 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:35.999124 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:35.999280 (+ 156us) server_negotiation.cc:299] Negotiation successful
0618 23:31:35.999417 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":52}
I20260618 23:31:36.002236 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45686:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260618 23:31:36.002497 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:31:36.004434 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac47348958a4798a9627d994928c6b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63c2b4767f314286912a649d5000765c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 23:31:36.004429 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac47348958a4798a9627d994928c6b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63c2b4767f314286912a649d5000765c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 23:31:36.004429 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 2ac47348958a4798a9627d994928c6b7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63c2b4767f314286912a649d5000765c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 23:31:36.004629 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac47348958a4798a9627d994928c6b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:36.004637 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac47348958a4798a9627d994928c6b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:36.004637 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ac47348958a4798a9627d994928c6b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:36.006148 2014 tablet_bootstrap.cc:492] T 2ac47348958a4798a9627d994928c6b7 P 7f1309bc848a44b1808e1e740c132d27: Bootstrap starting.
I20260618 23:31:36.006431 2017 tablet_bootstrap.cc:492] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2: Bootstrap starting.
I20260618 23:31:36.006440 2013 tablet_bootstrap.cc:492] T 2ac47348958a4798a9627d994928c6b7 P 51c9ab34b4cf46ba852e7d848e0a11c4: Bootstrap starting.
I20260618 23:31:36.006609 2014 tablet_bootstrap.cc:654] T 2ac47348958a4798a9627d994928c6b7 P 7f1309bc848a44b1808e1e740c132d27: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:36.006803 2013 tablet_bootstrap.cc:654] T 2ac47348958a4798a9627d994928c6b7 P 51c9ab34b4cf46ba852e7d848e0a11c4: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:36.007071 2017 tablet_bootstrap.cc:654] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:36.007438 2013 tablet_bootstrap.cc:492] T 2ac47348958a4798a9627d994928c6b7 P 51c9ab34b4cf46ba852e7d848e0a11c4: No bootstrap required, opened a new log
I20260618 23:31:36.007524 2013 ts_tablet_manager.cc:1397] T 2ac47348958a4798a9627d994928c6b7 P 51c9ab34b4cf46ba852e7d848e0a11c4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:36.007750 2013 raft_consensus.cc:348] T 2ac47348958a4798a9627d994928c6b7 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } }
I20260618 23:31:36.007850 2013 raft_consensus.cc:374] T 2ac47348958a4798a9627d994928c6b7 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:36.007869 2013 raft_consensus.cc:729] T 2ac47348958a4798a9627d994928c6b7 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51c9ab34b4cf46ba852e7d848e0a11c4, State: Initialized, Role: FOLLOWER
I20260618 23:31:36.007916 2017 tablet_bootstrap.cc:492] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2: No bootstrap required, opened a new log
I20260618 23:31:36.007932 2013 consensus_queue.cc:260] T 2ac47348958a4798a9627d994928c6b7 P 51c9ab34b4cf46ba852e7d848e0a11c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } }
I20260618 23:31:36.007966 2017 ts_tablet_manager.cc:1397] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:31:36.008056 2013 ts_tablet_manager.cc:1428] T 2ac47348958a4798a9627d994928c6b7 P 51c9ab34b4cf46ba852e7d848e0a11c4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:36.008082 2014 tablet_bootstrap.cc:492] T 2ac47348958a4798a9627d994928c6b7 P 7f1309bc848a44b1808e1e740c132d27: No bootstrap required, opened a new log
I20260618 23:31:36.008131 2014 ts_tablet_manager.cc:1397] T 2ac47348958a4798a9627d994928c6b7 P 7f1309bc848a44b1808e1e740c132d27: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260618 23:31:36.008065 2034 tablet_replica.cc:439] T 2ac47348958a4798a9627d994928c6b7 P 51c9ab34b4cf46ba852e7d848e0a11c4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } } }
I20260618 23:31:36.008193 2034 tablet_replica.cc:442] T 2ac47348958a4798a9627d994928c6b7 P 51c9ab34b4cf46ba852e7d848e0a11c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:31:36.008150 2017 raft_consensus.cc:348] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } }
I20260618 23:31:36.008253 2017 raft_consensus.cc:374] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:36.008275 2017 raft_consensus.cc:729] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 233d75d48e4b465381298a1f2e2ff5d2, State: Initialized, Role: FOLLOWER
I20260618 23:31:36.008328 2017 consensus_queue.cc:260] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } }
I20260618 23:31:36.008304 2014 raft_consensus.cc:348] T 2ac47348958a4798a9627d994928c6b7 P 7f1309bc848a44b1808e1e740c132d27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } }
I20260618 23:31:36.008373 2014 raft_consensus.cc:374] T 2ac47348958a4798a9627d994928c6b7 P 7f1309bc848a44b1808e1e740c132d27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:36.008394 2014 raft_consensus.cc:729] T 2ac47348958a4798a9627d994928c6b7 P 7f1309bc848a44b1808e1e740c132d27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f1309bc848a44b1808e1e740c132d27, State: Initialized, Role: FOLLOWER
I20260618 23:31:36.008442 2017 ts_tablet_manager.cc:1428] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:36.008440 2014 consensus_queue.cc:260] T 2ac47348958a4798a9627d994928c6b7 P 7f1309bc848a44b1808e1e740c132d27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } }
I20260618 23:31:36.008538 2014 ts_tablet_manager.cc:1428] T 2ac47348958a4798a9627d994928c6b7 P 7f1309bc848a44b1808e1e740c132d27: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:36.008535 2021 tablet_replica.cc:439] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } } }
I20260618 23:31:36.008690 2021 tablet_replica.cc:442] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:31:36.008769 2018 tablet_replica.cc:439] T 2ac47348958a4798a9627d994928c6b7 P 7f1309bc848a44b1808e1e740c132d27: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } } }
I20260618 23:31:36.008850 2018 tablet_replica.cc:442] T 2ac47348958a4798a9627d994928c6b7 P 7f1309bc848a44b1808e1e740c132d27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:31:36.022691 2021 raft_consensus.cc:482] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:31:36.022853 2021 raft_consensus.cc:504] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } }
I20260618 23:31:36.023257 2021 leader_election.cc:288] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f1309bc848a44b1808e1e740c132d27 (127.0.98.1:45933), 51c9ab34b4cf46ba852e7d848e0a11c4 (127.0.98.3:32927)
I20260618 23:31:36.026425 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:36.023606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42013 (local address 127.0.98.3:32927)
0618 23:31:36.023648 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:36.023651 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:36.023820 (+ 169us) server_negotiation.cc:407] Connection header received
0618 23:31:36.023899 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:36.023900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:36.023913 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:36.023971 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:36.024125 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.024630 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.025681 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.026069 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.026156 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:36.026166 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:36.026315 (+ 149us) server_negotiation.cc:299] Negotiation successful
0618 23:31:36.026353 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260618 23:31:36.026453 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:36.023503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32927 (local address 127.0.98.2:42013)
0618 23:31:36.023735 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:36.023747 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:36.023843 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:36.023975 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:36.023976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:36.023978 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:36.024047 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:36.024050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.024719 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.024721 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:36.025588 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:36.025590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.026157 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.026158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:36.026230 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:36.026235 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:31:36.026311 (+ 76us) client_negotiation.cc:240] Negotiation successful
0618 23:31:36.026359 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":1,"thread_start_us":150,"threads_started":1}
I20260618 23:31:36.026638 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:36.023478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41223 (local address 127.0.98.1:45933)
0618 23:31:36.023530 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:36.023533 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:36.023830 (+ 297us) server_negotiation.cc:407] Connection header received
0618 23:31:36.023908 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:36.023910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:36.023922 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:36.024002 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:36.024208 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.024742 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.025804 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.026201 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.026289 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:36.026301 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:36.026466 (+ 165us) server_negotiation.cc:299] Negotiation successful
0618 23:31:36.026510 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260618 23:31:36.027034 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:36.023424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45933 (local address 127.0.98.2:41223)
0618 23:31:36.023722 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:36.023736 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:36.023846 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:36.024004 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:36.024006 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:36.024009 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:36.024131 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:36.024134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.024834 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.024835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:36.025704 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:36.025706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.026288 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.026289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:36.026366 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:36.026372 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:31:36.026458 (+ 86us) client_negotiation.cc:240] Negotiation successful
0618 23:31:36.026935 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":76,"threads_started":1}
I20260618 23:31:36.026968 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac47348958a4798a9627d994928c6b7" candidate_uuid: "233d75d48e4b465381298a1f2e2ff5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" is_pre_election: true
I20260618 23:31:36.027103 1944 raft_consensus.cc:2454] T 2ac47348958a4798a9627d994928c6b7 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 233d75d48e4b465381298a1f2e2ff5d2 in term 0.
I20260618 23:31:36.027328 1735 leader_election.cc:302] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 233d75d48e4b465381298a1f2e2ff5d2, 51c9ab34b4cf46ba852e7d848e0a11c4; no voters:
I20260618 23:31:36.027323 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac47348958a4798a9627d994928c6b7" candidate_uuid: "233d75d48e4b465381298a1f2e2ff5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f1309bc848a44b1808e1e740c132d27" is_pre_election: true
I20260618 23:31:36.027434 1664 raft_consensus.cc:2454] T 2ac47348958a4798a9627d994928c6b7 P 7f1309bc848a44b1808e1e740c132d27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 233d75d48e4b465381298a1f2e2ff5d2 in term 0.
I20260618 23:31:36.027489 2021 raft_consensus.cc:2781] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:31:36.027532 2021 raft_consensus.cc:482] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:31:36.027619 2021 raft_consensus.cc:3037] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:36.028587 2021 raft_consensus.cc:504] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } }
I20260618 23:31:36.028771 2021 leader_election.cc:288] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2 [CANDIDATE]: Term 1 election: Requested vote from peers 7f1309bc848a44b1808e1e740c132d27 (127.0.98.1:45933), 51c9ab34b4cf46ba852e7d848e0a11c4 (127.0.98.3:32927)
I20260618 23:31:36.028983 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac47348958a4798a9627d994928c6b7" candidate_uuid: "233d75d48e4b465381298a1f2e2ff5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f1309bc848a44b1808e1e740c132d27"
I20260618 23:31:36.029093 1664 raft_consensus.cc:3037] T 2ac47348958a4798a9627d994928c6b7 P 7f1309bc848a44b1808e1e740c132d27 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:36.029093 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ac47348958a4798a9627d994928c6b7" candidate_uuid: "233d75d48e4b465381298a1f2e2ff5d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4"
I20260618 23:31:36.029201 1944 raft_consensus.cc:3037] T 2ac47348958a4798a9627d994928c6b7 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:36.029846 1664 raft_consensus.cc:2454] T 2ac47348958a4798a9627d994928c6b7 P 7f1309bc848a44b1808e1e740c132d27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 233d75d48e4b465381298a1f2e2ff5d2 in term 1.
I20260618 23:31:36.029943 1944 raft_consensus.cc:2454] T 2ac47348958a4798a9627d994928c6b7 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 233d75d48e4b465381298a1f2e2ff5d2 in term 1.
I20260618 23:31:36.030046 1736 leader_election.cc:302] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 233d75d48e4b465381298a1f2e2ff5d2, 7f1309bc848a44b1808e1e740c132d27; no voters:
I20260618 23:31:36.030148 2021 raft_consensus.cc:2781] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:36.030314 2021 raft_consensus.cc:686] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2 [term 1 LEADER]: Becoming Leader. State: Replica: 233d75d48e4b465381298a1f2e2ff5d2, State: Running, Role: LEADER
I20260618 23:31:36.030422 2021 consensus_queue.cc:237] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } }
I20260618 23:31:36.030915 2057 tablet_replica.cc:439] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2: TxnStatusTablet state changed. Reason: New leader 233d75d48e4b465381298a1f2e2ff5d2. Latest consensus state: current_term: 1 leader_uuid: "233d75d48e4b465381298a1f2e2ff5d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } } }
I20260618 23:31:36.031003 2057 tablet_replica.cc:442] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2: This TxnStatusTablet replica's current role is: LEADER
I20260618 23:31:36.031194 2059 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260618 23:31:36.031325 2059 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260618 23:31:36.031317 1506 catalog_manager.cc:5495] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 233d75d48e4b465381298a1f2e2ff5d2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "233d75d48e4b465381298a1f2e2ff5d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } health_report { overall_health: UNKNOWN } } }
I20260618 23:31:36.041421 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:36.038286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55838 (local address 127.0.98.2:38663)
0618 23:31:36.038369 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:36.038372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:36.038403 (+ 31us) server_negotiation.cc:407] Connection header received
0618 23:31:36.038436 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:36.038437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:36.038459 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:36.038540 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:31:36.038744 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.039499 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.040424 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.040728 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.040810 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:36.041033 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:31:36.041162 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:31:36.041248 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:36.041310 (+ 62us) server_negotiation.cc:299] Negotiation successful
0618 23:31:36.041331 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 23:31:36.041430 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:36.038140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38663 (local address 127.0.0.1:55838)
0618 23:31:36.038255 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:36.038272 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:36.038345 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:36.038593 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:36.038595 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:36.038596 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:31:36.038658 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:36.038661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.039642 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.039643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:36.040309 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:36.040311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.040839 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.040841 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:36.040902 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:36.040919 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:31:36.041247 (+ 328us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:31:36.041253 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:31:36.041302 (+ 49us) client_negotiation.cc:240] Negotiation successful
0618 23:31:36.041356 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260618 23:31:36.042894 1664 raft_consensus.cc:1261] T 2ac47348958a4798a9627d994928c6b7 P 7f1309bc848a44b1808e1e740c132d27 [term 1 FOLLOWER]: Refusing update from remote peer 233d75d48e4b465381298a1f2e2ff5d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:31:36.042964 1944 raft_consensus.cc:1261] T 2ac47348958a4798a9627d994928c6b7 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 1 FOLLOWER]: Refusing update from remote peer 233d75d48e4b465381298a1f2e2ff5d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:31:36.043221 2057 consensus_queue.cc:1035] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:31:36.043409 2021 consensus_queue.cc:1035] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:31:36.044363 2034 tablet_replica.cc:439] T 2ac47348958a4798a9627d994928c6b7 P 51c9ab34b4cf46ba852e7d848e0a11c4: TxnStatusTablet state changed. Reason: New leader 233d75d48e4b465381298a1f2e2ff5d2. Latest consensus state: current_term: 1 leader_uuid: "233d75d48e4b465381298a1f2e2ff5d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } } }
I20260618 23:31:36.044457 2034 tablet_replica.cc:442] T 2ac47348958a4798a9627d994928c6b7 P 51c9ab34b4cf46ba852e7d848e0a11c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:31:36.044721 2018 tablet_replica.cc:439] T 2ac47348958a4798a9627d994928c6b7 P 7f1309bc848a44b1808e1e740c132d27: TxnStatusTablet state changed. Reason: New leader 233d75d48e4b465381298a1f2e2ff5d2. Latest consensus state: current_term: 1 leader_uuid: "233d75d48e4b465381298a1f2e2ff5d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } } }
I20260618 23:31:36.044812 2018 tablet_replica.cc:442] T 2ac47348958a4798a9627d994928c6b7 P 7f1309bc848a44b1808e1e740c132d27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:31:36.045269 2057 tablet_replica.cc:439] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "233d75d48e4b465381298a1f2e2ff5d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } } }
I20260618 23:31:36.045369 2057 tablet_replica.cc:442] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2: This TxnStatusTablet replica's current role is: LEADER
I20260618 23:31:36.046036 2026 mvcc.cc:205] Tried to move back new op lower bound from 7298357231789039616 to 7298357231741411328. Current Snapshot: MvccSnapshot[applied={T|T < 7298357231789039616 or (T in {7298357231789039616})}]
I20260618 23:31:36.045552 2057 tablet_replica.cc:439] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "233d75d48e4b465381298a1f2e2ff5d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } } }
I20260618 23:31:36.046119 2057 tablet_replica.cc:442] T 2ac47348958a4798a9627d994928c6b7 P 233d75d48e4b465381298a1f2e2ff5d2: This TxnStatusTablet replica's current role is: LEADER
I20260618 23:31:36.046195 2034 tablet_replica.cc:439] T 2ac47348958a4798a9627d994928c6b7 P 51c9ab34b4cf46ba852e7d848e0a11c4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "233d75d48e4b465381298a1f2e2ff5d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } } }
I20260618 23:31:36.046284 2034 tablet_replica.cc:442] T 2ac47348958a4798a9627d994928c6b7 P 51c9ab34b4cf46ba852e7d848e0a11c4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:31:36.046481 2018 tablet_replica.cc:439] T 2ac47348958a4798a9627d994928c6b7 P 7f1309bc848a44b1808e1e740c132d27: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "233d75d48e4b465381298a1f2e2ff5d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f1309bc848a44b1808e1e740c132d27" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45933 } } peers { permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 } } peers { permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 } } }
I20260618 23:31:36.046564 2018 tablet_replica.cc:442] T 2ac47348958a4798a9627d994928c6b7 P 7f1309bc848a44b1808e1e740c132d27: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:31:36.051491 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:36.048201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39284 (local address 127.0.98.1:45933)
0618 23:31:36.048272 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:36.048274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:36.048291 (+ 17us) server_negotiation.cc:407] Connection header received
0618 23:31:36.048334 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:36.048335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:36.048357 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:36.048449 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:31:36.048657 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.049423 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.050367 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.050681 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.050757 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:36.051073 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:31:36.051201 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:31:36.051315 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:36.051361 (+ 46us) server_negotiation.cc:299] Negotiation successful
0618 23:31:36.051389 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 23:31:36.058056 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:36.054759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41305 (local address 127.0.98.2:38663)
0618 23:31:36.054818 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:36.054820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:36.054965 (+ 145us) server_negotiation.cc:407] Connection header received
0618 23:31:36.054996 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:36.054997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:36.055007 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:36.055080 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:31:36.055298 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.056077 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.057139 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.057457 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.057547 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:36.057756 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:31:36.057792 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:31:36.057855 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:36.057948 (+ 93us) server_negotiation.cc:299] Negotiation successful
0618 23:31:36.057970 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260618 23:31:36.058063 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:36.054695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38663 (local address 127.0.98.1:41305)
0618 23:31:36.054871 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:36.054883 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:36.054953 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:36.055115 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:36.055116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:36.055118 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:31:36.055201 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:36.055202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.056185 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.056187 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:36.057002 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:36.057005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.057567 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.057568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:36.057629 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:36.057650 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:31:36.057873 (+ 223us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:31:36.057875 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:31:36.057921 (+ 46us) client_negotiation.cc:240] Negotiation successful
0618 23:31:36.057967 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":103,"threads_started":1}
I20260618 23:31:36.061307 1944 raft_consensus.cc:1261] T 2cee7c153e55483097f52cc5e2cce339 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 1 FOLLOWER]: Refusing update from remote peer 7f1309bc848a44b1808e1e740c132d27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:31:36.061307 1804 raft_consensus.cc:1261] T 2cee7c153e55483097f52cc5e2cce339 P 233d75d48e4b465381298a1f2e2ff5d2 [term 1 FOLLOWER]: Refusing update from remote peer 7f1309bc848a44b1808e1e740c132d27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:31:36.061626 2044 consensus_queue.cc:1035] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51c9ab34b4cf46ba852e7d848e0a11c4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:31:36.061774 2018 consensus_queue.cc:1035] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "233d75d48e4b465381298a1f2e2ff5d2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:31:36.074431 2072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:36.070488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45933 (local address 127.0.98.2:60825)
0618 23:31:36.070712 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:36.070725 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:36.070803 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:36.071504 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:36.071505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:36.071508 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:31:36.071602 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:36.071605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.072531 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.072532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:36.073515 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:36.073517 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.073913 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.073913 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:36.073981 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:36.074010 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:31:36.074224 (+ 214us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:31:36.074226 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:31:36.074265 (+ 39us) client_negotiation.cc:240] Negotiation successful
0618 23:31:36.074322 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":133,"threads_started":1}
I20260618 23:31:36.074409 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:36.071333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60825 (local address 127.0.98.1:45933)
0618 23:31:36.071382 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:36.071384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:36.071390 (+ 6us) server_negotiation.cc:407] Connection header received
0618 23:31:36.071412 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:36.071412 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:36.071422 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:36.071485 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:31:36.071664 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.072444 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.073611 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.073827 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.073912 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:36.074111 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:31:36.074149 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:31:36.074199 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:36.074281 (+ 82us) server_negotiation.cc:299] Negotiation successful
0618 23:31:36.074304 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260618 23:31:36.086974 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:31:36.087113 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:31:36.087888 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 75bae1580e4d450ab7fe5da074588771: Sending DeleteTablet for 3 replicas of tablet 2cee7c153e55483097f52cc5e2cce339
I20260618 23:31:36.088189 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cee7c153e55483097f52cc5e2cce339 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47158
I20260618 23:31:36.088313 2037 tablet_replica.cc:330] T 2cee7c153e55483097f52cc5e2cce339 P 51c9ab34b4cf46ba852e7d848e0a11c4: stopping tablet replica
I20260618 23:31:36.088376 2037 raft_consensus.cc:2229] T 2cee7c153e55483097f52cc5e2cce339 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:31:36.088424 2037 raft_consensus.cc:2258] T 2cee7c153e55483097f52cc5e2cce339 P 51c9ab34b4cf46ba852e7d848e0a11c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:36.088707 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cee7c153e55483097f52cc5e2cce339 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39274
I20260618 23:31:36.088829 2036 tablet_replica.cc:330] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27: stopping tablet replica
I20260618 23:31:36.088902 2036 raft_consensus.cc:2229] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:31:36.089030 2036 raft_consensus.cc:2258] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:36.089092 2037 ts_tablet_manager.cc:1905] T 2cee7c153e55483097f52cc5e2cce339 P 51c9ab34b4cf46ba852e7d848e0a11c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:31:36.089468 2036 ts_tablet_manager.cc:1905] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:31:36.090731 2037 ts_tablet_manager.cc:1918] T 2cee7c153e55483097f52cc5e2cce339 P 51c9ab34b4cf46ba852e7d848e0a11c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:31:36.090790 2037 log.cc:1196] T 2cee7c153e55483097f52cc5e2cce339 P 51c9ab34b4cf46ba852e7d848e0a11c4: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/2cee7c153e55483097f52cc5e2cce339
I20260618 23:31:36.091012 2037 ts_tablet_manager.cc:1939] T 2cee7c153e55483097f52cc5e2cce339 P 51c9ab34b4cf46ba852e7d848e0a11c4: Deleting consensus metadata
I20260618 23:31:36.091353 1491 catalog_manager.cc:4841] TS 51c9ab34b4cf46ba852e7d848e0a11c4 (127.0.98.3:32927): tablet 2cee7c153e55483097f52cc5e2cce339 (table test-table [id=c7bab42c4fe344a7baf297a86f866378]) successfully deleted
I20260618 23:31:36.091413 2036 ts_tablet_manager.cc:1918] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:31:36.091480 2036 log.cc:1196] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-0/wal/wals/2cee7c153e55483097f52cc5e2cce339
I20260618 23:31:36.091508 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cee7c153e55483097f52cc5e2cce339 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55828
I20260618 23:31:36.091629 2038 tablet_replica.cc:330] T 2cee7c153e55483097f52cc5e2cce339 P 233d75d48e4b465381298a1f2e2ff5d2: stopping tablet replica
I20260618 23:31:36.091686 2038 raft_consensus.cc:2229] T 2cee7c153e55483097f52cc5e2cce339 P 233d75d48e4b465381298a1f2e2ff5d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:31:36.091723 2038 raft_consensus.cc:2258] T 2cee7c153e55483097f52cc5e2cce339 P 233d75d48e4b465381298a1f2e2ff5d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:36.092032 2036 ts_tablet_manager.cc:1939] T 2cee7c153e55483097f52cc5e2cce339 P 7f1309bc848a44b1808e1e740c132d27: Deleting consensus metadata
I20260618 23:31:36.092461 1492 catalog_manager.cc:4841] TS 7f1309bc848a44b1808e1e740c132d27 (127.0.98.1:45933): tablet 2cee7c153e55483097f52cc5e2cce339 (table test-table [id=c7bab42c4fe344a7baf297a86f866378]) successfully deleted
I20260618 23:31:36.092782 2038 ts_tablet_manager.cc:1905] T 2cee7c153e55483097f52cc5e2cce339 P 233d75d48e4b465381298a1f2e2ff5d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:31:36.094372 2038 ts_tablet_manager.cc:1918] T 2cee7c153e55483097f52cc5e2cce339 P 233d75d48e4b465381298a1f2e2ff5d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:31:36.094440 2038 log.cc:1196] T 2cee7c153e55483097f52cc5e2cce339 P 233d75d48e4b465381298a1f2e2ff5d2: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/2cee7c153e55483097f52cc5e2cce339
I20260618 23:31:36.094653 2038 ts_tablet_manager.cc:1939] T 2cee7c153e55483097f52cc5e2cce339 P 233d75d48e4b465381298a1f2e2ff5d2: Deleting consensus metadata
I20260618 23:31:36.095049 1491 catalog_manager.cc:4841] TS 233d75d48e4b465381298a1f2e2ff5d2 (127.0.98.2:38663): tablet 2cee7c153e55483097f52cc5e2cce339 (table test-table [id=c7bab42c4fe344a7baf297a86f866378]) successfully deleted
Jun 18 23:31:36 dist-test-slave-6224 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825495, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260618 23:31:36.107897 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:36.095909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39290 (local address 127.0.98.1:45933)
0618 23:31:36.095970 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:36.095973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:36.096205 (+ 232us) server_negotiation.cc:407] Connection header received
0618 23:31:36.096292 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:36.096293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:36.096315 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:36.096380 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:36.097230 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.097973 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.099164 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.099512 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.099654 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:36.104170 (+ 4516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:36.104183 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:36.104183 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:36.104192 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:36.106417 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:36.106836 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:36.106840 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:36.106840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:36.106898 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:36.107090 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:36.107091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:36.107091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:36.107247 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:36.107401 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:36.107679 (+ 278us) server_negotiation.cc:299] Negotiation successful
0618 23:31:36.107763 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260618 23:31:36.108491 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39290
I20260618 23:31:36.117704 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:36.109797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45690 (local address 127.0.98.62:33725)
0618 23:31:36.109848 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:36.109849 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:36.109953 (+ 104us) server_negotiation.cc:407] Connection header received
0618 23:31:36.110026 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:36.110027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:36.110040 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:36.110101 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:36.110821 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.111660 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.112737 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:36.112992 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:36.113039 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:36.114640 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:36.114641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:36.114642 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:36.114648 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:36.116384 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:36.116830 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:36.116831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:36.116831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:36.116899 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:36.117098 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:36.117099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:36.117099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:36.117217 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:36.117292 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:36.117508 (+ 216us) server_negotiation.cc:299] Negotiation successful
0618 23:31:36.117606 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260618 23:31:36.118472 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45690
I20260618 23:31:36.119678 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 1579
I20260618 23:31:36.126788 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 1719
I20260618 23:31:36.133360 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 1859
I20260618 23:31:36.139834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 1475
2026-06-18T23:31:36Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2327 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:31:36 dist-test-slave-6224 krb5kdc[2081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:31:36 dist-test-slave-6224 krb5kdc[2081](info): set up 2 sockets
Jun 18 23:31:36 dist-test-slave-6224 krb5kdc[2081](info): commencing operation
krb5kdc: starting...
W20260618 23:31:37.179455 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:31:37 dist-test-slave-6224 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:31:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:31:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:31:37.311061 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44909
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44909
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:37.406213 2094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:37.406477 2094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:37.406512 2094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:37.407972 2094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:31:37.408036 2094 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260618 23:31:37.408051 2094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:37.408064 2094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:37.408077 2094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:31:37.408092 2094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:31:37.409852 2094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44909
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:37.410168 2094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:37.410451 2094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:37.416064 2102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:37.416064 2099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:37.416139 2100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:37.416519 2094 server_base.cc:1034] running on GCE node
I20260618 23:31:37.416841 2094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:37.417196 2094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:37.418357 2094 hybrid_clock.cc:648] HybridClock initialized: now 1781825497418331 us; error 42 us; skew 500 ppm
Jun 18 23:31:37 dist-test-slave-6224 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:37.421260 2094 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:31:37.421828 2094 webserver.cc:460] Webserver started at http://127.0.98.62:36559/ using document root <none> and password file <none>
I20260618 23:31:37.422104 2094 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:37.422181 2094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:37.422339 2094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:37.423457 2094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "a6a6c7c3b43649378d624394efbe9918"
format_stamp: "Formatted at 2026-06-18 23:31:37 on dist-test-slave-6224"
server_key: "c9ede1911cde8a0424c2e9ba59acbf1d"
server_key_iv: "9874530e1160d392d6d58b0cb5ec8697"
server_key_version: "encryptionkey@0"
I20260618 23:31:37.423933 2094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "a6a6c7c3b43649378d624394efbe9918"
format_stamp: "Formatted at 2026-06-18 23:31:37 on dist-test-slave-6224"
server_key: "c9ede1911cde8a0424c2e9ba59acbf1d"
server_key_iv: "9874530e1160d392d6d58b0cb5ec8697"
server_key_version: "encryptionkey@0"
I20260618 23:31:37.425539 2094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:37.426585 2109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:37.426985 2094 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:37.427094 2094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "a6a6c7c3b43649378d624394efbe9918"
format_stamp: "Formatted at 2026-06-18 23:31:37 on dist-test-slave-6224"
server_key: "c9ede1911cde8a0424c2e9ba59acbf1d"
server_key_iv: "9874530e1160d392d6d58b0cb5ec8697"
server_key_version: "encryptionkey@0"
I20260618 23:31:37.427186 2094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:37.447758 2094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:37.450243 2094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:37.450423 2094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:37.456149 2094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44909
I20260618 23:31:37.456198 2171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44909 every 8 connection(s)
I20260618 23:31:37.456573 2094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:31:37.457299 2172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:37.457340 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 2094
I20260618 23:31:37.457485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:31:37.457732 392 external_mini_cluster.cc:1428] Setting key e3c7cbbb36f4a02e0ee8c39073869537
I20260618 23:31:37.460534 2172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918: Bootstrap starting.
I20260618 23:31:37.461256 2172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:37.461578 2172 log.cc:824] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:37.462724 2172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918: No bootstrap required, opened a new log
I20260618 23:31:37.465071 2172 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a6c7c3b43649378d624394efbe9918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44909 } }
I20260618 23:31:37.465229 2172 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:37.465252 2172 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6a6c7c3b43649378d624394efbe9918, State: Initialized, Role: FOLLOWER
I20260618 23:31:37.465402 2172 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a6c7c3b43649378d624394efbe9918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44909 } }
I20260618 23:31:37.465484 2172 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:31:37.465515 2172 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:31:37.465560 2172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:37.466166 2172 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a6c7c3b43649378d624394efbe9918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44909 } }
I20260618 23:31:37.466303 2172 leader_election.cc:302] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6a6c7c3b43649378d624394efbe9918; no voters:
I20260618 23:31:37.466542 2172 leader_election.cc:288] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:31:37.466580 2177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:37.466692 2177 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918 [term 1 LEADER]: Becoming Leader. State: Replica: a6a6c7c3b43649378d624394efbe9918, State: Running, Role: LEADER
Jun 18 23:31:37 dist-test-slave-6224 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825497, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:37.466785 2177 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a6c7c3b43649378d624394efbe9918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44909 } }
I20260618 23:31:37.466851 2172 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:31:37.467152 2177 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6a6c7c3b43649378d624394efbe9918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a6c7c3b43649378d624394efbe9918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44909 } } }
I20260618 23:31:37.467235 2177 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918 [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:37.467590 2177 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6a6c7c3b43649378d624394efbe9918. Latest consensus state: current_term: 1 leader_uuid: "a6a6c7c3b43649378d624394efbe9918" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6a6c7c3b43649378d624394efbe9918" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44909 } } }
I20260618 23:31:37.467705 2177 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918 [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:37.468638 2182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:31:37.469523 2182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:31:37.471338 2182 catalog_manager.cc:1347] Generated new cluster ID: dc3fdff0e21f4631a5906dadb9d5d163
I20260618 23:31:37.471415 2182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:31:37.472500 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:37.458484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57296 (local address 127.0.98.62:44909)
0618 23:31:37.458746 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:37.458757 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:37.458800 (+ 43us) server_negotiation.cc:407] Connection header received
0618 23:31:37.459355 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:37.459365 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:37.459430 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:37.459656 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:37.460371 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:37.461780 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:37.463123 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:37.463793 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:37.463932 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:37.467529 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:37.467537 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:37.467537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:37.467583 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:37.470581 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:37.471122 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:37.471126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:37.471126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:37.471277 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:37.471524 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:37.471525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:37.471525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:37.471839 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:37.472011 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:37.472129 (+ 118us) server_negotiation.cc:299] Negotiation successful
0618 23:31:37.472289 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":123,"threads_started":1}
I20260618 23:31:37.518462 2182 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:31:37.519276 2182 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:31:37.524914 2182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918: Generated new TSK 0
I20260618 23:31:37.525126 2182 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260618 23:31:37.558116 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44909
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:37.645174 2198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:37.645404 2198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:37.645431 2198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:37.646621 2198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:37.646679 2198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:37.646700 2198 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260618 23:31:37.646713 2198 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 23:31:37.646729 2198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:31:37.648304 2198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:37.648562 2198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:37.648792 2198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:37.654121 2203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:37.654137 2204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:37.654332 2198 server_base.cc:1034] running on GCE node
W20260618 23:31:37.654137 2206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:37.654734 2198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:37.655049 2198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:37.656216 2198 hybrid_clock.cc:648] HybridClock initialized: now 1781825497656177 us; error 60 us; skew 500 ppm
Jun 18 23:31:37 dist-test-slave-6224 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:37.659058 2198 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:31:37.659616 2198 webserver.cc:460] Webserver started at http://127.0.98.1:37301/ using document root <none> and password file <none>
I20260618 23:31:37.659871 2198 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:37.659942 2198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:37.660095 2198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:37.661222 2198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "c05450364e9e4578a1fa7034f7c23b2b"
format_stamp: "Formatted at 2026-06-18 23:31:37 on dist-test-slave-6224"
server_key: "bd211e735c9197fe023b442b5d53155d"
server_key_iv: "cdea5ed306b1e4b04b79b6aea61d5296"
server_key_version: "encryptionkey@0"
I20260618 23:31:37.661593 2198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c05450364e9e4578a1fa7034f7c23b2b"
format_stamp: "Formatted at 2026-06-18 23:31:37 on dist-test-slave-6224"
server_key: "bd211e735c9197fe023b442b5d53155d"
server_key_iv: "cdea5ed306b1e4b04b79b6aea61d5296"
server_key_version: "encryptionkey@0"
I20260618 23:31:37.663090 2198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:31:37.664143 2213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:37.664413 2198 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260618 23:31:37.664538 2198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "c05450364e9e4578a1fa7034f7c23b2b"
format_stamp: "Formatted at 2026-06-18 23:31:37 on dist-test-slave-6224"
server_key: "bd211e735c9197fe023b442b5d53155d"
server_key_iv: "cdea5ed306b1e4b04b79b6aea61d5296"
server_key_version: "encryptionkey@0"
I20260618 23:31:37.664640 2198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:37.674815 2198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:37.677369 2198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:37.677580 2198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:37.678313 2198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:37.678391 2198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:37.678438 2198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:37.678488 2198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 23:31:37 dist-test-slave-6224 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:37.689307 2198 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42811
I20260618 23:31:37.689424 2328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42811 every 8 connection(s)
I20260618 23:31:37.689889 2198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:31:37.692612 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:37.678979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54721 (local address 127.0.98.62:44909)
0618 23:31:37.679071 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:37.679074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:37.679284 (+ 210us) server_negotiation.cc:407] Connection header received
0618 23:31:37.680168 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:37.680170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:37.680188 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:37.680293 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:37.681712 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:37.682533 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:37.684617 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:37.684969 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:37.685083 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:37.689212 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:37.689213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:37.689214 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:37.689223 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:37.691083 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:37.691544 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:37.691567 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:37.691568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:37.691628 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:37.691909 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:37.691910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:37.691910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:37.692046 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:37.692118 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:37.692427 (+ 309us) server_negotiation.cc:299] Negotiation successful
0618 23:31:37.692515 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 23:31:37.692860 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:37.678949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44909 (local address 127.0.98.1:54721)
0618 23:31:37.679156 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:37.679200 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:37.679924 (+ 724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:37.680488 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:37.680496 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:37.681027 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:37.681539 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:37.681551 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:37.682682 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:37.682684 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:37.684481 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:37.684485 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:37.685116 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:37.685117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:37.685451 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:37.686464 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:37.686480 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:37.689082 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:37.691187 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:37.691195 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:37.691198 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:37.691456 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:37.691704 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:37.691705 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:37.691706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:37.691833 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:37.692120 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:37.692127 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:37.692285 (+ 158us) client_negotiation.cc:769] Sending connection context
0618 23:31:37.692391 (+ 106us) client_negotiation.cc:240] Negotiation successful
0618 23:31:37.692582 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":138,"threads_started":1}
I20260618 23:31:37.693557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 2198
I20260618 23:31:37.693619 2329 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44909
I20260618 23:31:37.693691 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:31:37.693744 2329 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:37.693922 392 external_mini_cluster.cc:1428] Setting key 970b345976bbbdd428116e0177793f77
I20260618 23:31:37.693948 2329 heartbeater.cc:511] Master 127.0.98.62:44909 requested a full tablet report, sending...
I20260618 23:31:37.694619 2125 ts_manager.cc:194] Registered new tserver with Master: c05450364e9e4578a1fa7034f7c23b2b (127.0.98.1:42811)
I20260618 23:31:37.695780 2125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54721
I20260618 23:31:37.707072 2335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:37.694980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44909 (local address 127.0.98.1:50019)
0618 23:31:37.695195 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:37.695244 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:37.695298 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:37.699199 (+ 3901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:37.699200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:37.699485 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:37.699612 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:37.699615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:37.700632 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:37.700634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:37.701759 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:37.701762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:37.702365 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:37.702367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:37.702452 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:37.703100 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:37.703106 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:37.703598 (+ 492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:37.705847 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:37.705848 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:37.705849 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:37.706088 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:37.706359 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:37.706360 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:37.706360 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:37.706422 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:37.706748 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:37.706749 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:37.706800 (+ 51us) client_negotiation.cc:769] Sending connection context
0618 23:31:37.706858 (+ 58us) client_negotiation.cc:240] Negotiation successful
0618 23:31:37.706970 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":127,"threads_started":1}
I20260618 23:31:37.707127 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:37.698676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50019 (local address 127.0.98.62:44909)
0618 23:31:37.698801 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:37.698803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:37.698814 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:31:37.698962 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:37.698963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:37.698977 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:37.699137 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:37.699721 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:37.700500 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:37.701876 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:37.702234 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:37.702347 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:37.703755 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:37.703756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:37.703756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:37.703765 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:37.705720 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:37.706188 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:37.706189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:37.706190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:37.706274 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:37.706500 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:37.706502 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:37.706502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:37.706668 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:37.706751 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:37.706913 (+ 162us) server_negotiation.cc:299] Negotiation successful
0618 23:31:37.707019 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260618 23:31:37.728638 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44909
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:37.809937 2338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:37.810153 2338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:37.810177 2338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:37.811267 2338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:37.811309 2338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:37.811331 2338 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260618 23:31:37.811343 2338 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 23:31:37.811360 2338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:31:37.812830 2338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:37.813030 2338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:37.813218 2338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:37.819116 2344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:37.819119 2346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:37.819098 2343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:37.819434 2338 server_base.cc:1034] running on GCE node
I20260618 23:31:37.819705 2338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:37.820011 2338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:37.821171 2338 hybrid_clock.cc:648] HybridClock initialized: now 1781825497821145 us; error 56 us; skew 500 ppm
Jun 18 23:31:37 dist-test-slave-6224 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:37.823930 2338 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:31:37.824431 2338 webserver.cc:460] Webserver started at http://127.0.98.2:40461/ using document root <none> and password file <none>
I20260618 23:31:37.824692 2338 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:37.824762 2338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:37.824940 2338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:37.826033 2338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "21e9603236b94e92b1695c9a1ed5c89d"
format_stamp: "Formatted at 2026-06-18 23:31:37 on dist-test-slave-6224"
server_key: "d3e8d756345c56ad2c1c5e9688e83878"
server_key_iv: "6166452622340045a6a3e569df715dbb"
server_key_version: "encryptionkey@0"
I20260618 23:31:37.826403 2338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "21e9603236b94e92b1695c9a1ed5c89d"
format_stamp: "Formatted at 2026-06-18 23:31:37 on dist-test-slave-6224"
server_key: "d3e8d756345c56ad2c1c5e9688e83878"
server_key_iv: "6166452622340045a6a3e569df715dbb"
server_key_version: "encryptionkey@0"
I20260618 23:31:37.827991 2338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260618 23:31:37.828991 2353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:37.829275 2338 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:37.829401 2338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "21e9603236b94e92b1695c9a1ed5c89d"
format_stamp: "Formatted at 2026-06-18 23:31:37 on dist-test-slave-6224"
server_key: "d3e8d756345c56ad2c1c5e9688e83878"
server_key_iv: "6166452622340045a6a3e569df715dbb"
server_key_version: "encryptionkey@0"
I20260618 23:31:37.829514 2338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:37.840610 2338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:37.842921 2338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:37.843116 2338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:37.843801 2338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:37.843870 2338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:37.843923 2338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:37.843952 2338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:37.851915 2338 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39127
I20260618 23:31:37.852034 2468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39127 every 8 connection(s)
Jun 18 23:31:37 dist-test-slave-6224 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:37.852406 2338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:31:37.854964 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 2338
I20260618 23:31:37.855083 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:31:37.855329 392 external_mini_cluster.cc:1428] Setting key f9c2fd7c1e767c87063674bca2c21252
I20260618 23:31:37.856776 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:37.844545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54293 (local address 127.0.98.62:44909)
0618 23:31:37.844618 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:37.844620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:37.844952 (+ 332us) server_negotiation.cc:407] Connection header received
0618 23:31:37.845664 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:37.845666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:37.845680 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:37.845729 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:37.846593 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:37.847472 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:37.848643 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:37.848874 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:37.848985 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:37.852951 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:37.852954 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:37.852954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:37.852990 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:37.854654 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:37.855387 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:37.855388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:37.855388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:37.855493 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:37.855942 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:37.855943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:37.855943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:37.856100 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:37.856191 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:37.856609 (+ 418us) server_negotiation.cc:299] Negotiation successful
0618 23:31:37.856696 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260618 23:31:37.857187 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:37.844545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44909 (local address 127.0.98.2:54293)
0618 23:31:37.844795 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:37.844844 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:37.845548 (+ 704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:37.845837 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:37.845860 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:37.846251 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:37.846488 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:37.846493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:37.847604 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:37.847605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:37.848535 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:37.848538 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:37.849042 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:37.849044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:37.849286 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:37.850615 (+ 1329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:37.850626 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:37.852821 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:37.854810 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:37.854820 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:37.854823 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:37.855235 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:37.855641 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:37.855642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:37.855642 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:37.855840 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:37.856201 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:37.856208 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:37.856420 (+ 212us) client_negotiation.cc:769] Sending connection context
0618 23:31:37.856583 (+ 163us) client_negotiation.cc:240] Negotiation successful
0618 23:31:37.856848 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":168,"threads_started":1}
I20260618 23:31:37.857846 2469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44909
I20260618 23:31:37.857963 2469 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:37.858218 2469 heartbeater.cc:511] Master 127.0.98.62:44909 requested a full tablet report, sending...
I20260618 23:31:37.858660 2125 ts_manager.cc:194] Registered new tserver with Master: 21e9603236b94e92b1695c9a1ed5c89d (127.0.98.2:39127)
I20260618 23:31:37.859124 2125 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54293
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260618 23:31:37.866047 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:37.859631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50379 (local address 127.0.98.62:44909)
0618 23:31:37.859674 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:37.859676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:37.859685 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:31:37.859753 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:37.859754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:37.859764 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:37.859818 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:37.860212 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:37.860698 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:37.861606 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:37.861791 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:37.861864 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:37.863122 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:37.863124 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:37.863124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:37.863131 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:37.864681 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:37.865132 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:37.865133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:37.865134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:37.865191 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:37.865406 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:37.865407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:37.865408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:37.865612 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:37.865697 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:37.865854 (+ 157us) server_negotiation.cc:299] Negotiation successful
0618 23:31:37.865937 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:31:37.866057 2476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:37.859432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44909 (local address 127.0.98.2:50379)
0618 23:31:37.859620 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:37.859633 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:37.859692 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:37.859833 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:37.859834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:37.860053 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:37.860120 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:37.860122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:37.860797 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:37.860798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:37.861509 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:37.861511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:37.861863 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:37.861864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:37.861938 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:37.862618 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:37.862623 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:37.863013 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:37.864842 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:37.864843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:37.864844 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:37.865040 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:37.865257 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:37.865258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:37.865258 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:37.865338 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:37.865688 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:37.865689 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:37.865738 (+ 49us) client_negotiation.cc:769] Sending connection context
0618 23:31:37.865830 (+ 92us) client_negotiation.cc:240] Negotiation successful
0618 23:31:37.865943 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260618 23:31:37.886505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44909
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:37.981987 2478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:37.982292 2478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:37.982334 2478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:37.983675 2478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:37.983762 2478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:37.983803 2478 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260618 23:31:37.983832 2478 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 23:31:37.983865 2478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:31:37.985566 2478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44909
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:37.985939 2478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:37.986239 2478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:37.992441 2484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:37.992444 2486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:37.992497 2483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:37.992800 2478 server_base.cc:1034] running on GCE node
I20260618 23:31:37.993074 2478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:37.993418 2478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:37.994621 2478 hybrid_clock.cc:648] HybridClock initialized: now 1781825497994587 us; error 54 us; skew 500 ppm
Jun 18 23:31:37 dist-test-slave-6224 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:37.997782 2478 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:31:37.998375 2478 webserver.cc:460] Webserver started at http://127.0.98.3:38835/ using document root <none> and password file <none>
I20260618 23:31:37.998648 2478 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:37.998723 2478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:37.998888 2478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:38.000422 2478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "faef863dbed84e0b927b5833deacec3d"
format_stamp: "Formatted at 2026-06-18 23:31:37 on dist-test-slave-6224"
server_key: "fc43f98a0a4fef4ed68887a75c2af5c1"
server_key_iv: "33ed88215de2fb3706725925cb866cdd"
server_key_version: "encryptionkey@0"
I20260618 23:31:38.000864 2478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "faef863dbed84e0b927b5833deacec3d"
format_stamp: "Formatted at 2026-06-18 23:31:37 on dist-test-slave-6224"
server_key: "fc43f98a0a4fef4ed68887a75c2af5c1"
server_key_iv: "33ed88215de2fb3706725925cb866cdd"
server_key_version: "encryptionkey@0"
I20260618 23:31:38.002822 2478 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260618 23:31:38.003957 2493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:38.004191 2478 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:38.004314 2478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "faef863dbed84e0b927b5833deacec3d"
format_stamp: "Formatted at 2026-06-18 23:31:37 on dist-test-slave-6224"
server_key: "fc43f98a0a4fef4ed68887a75c2af5c1"
server_key_iv: "33ed88215de2fb3706725925cb866cdd"
server_key_version: "encryptionkey@0"
I20260618 23:31:38.004403 2478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:38.016538 2478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:38.019155 2478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:38.019340 2478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:38.020174 2478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:38.020238 2478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:38.020285 2478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:38.020313 2478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 23:31:38 dist-test-slave-6224 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:38.029173 2478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45889
I20260618 23:31:38.029286 2608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45889 every 8 connection(s)
I20260618 23:31:38.029645 2478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:31:38.032074 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.020726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59557 (local address 127.0.98.62:44909)
0618 23:31:38.020812 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:38.020814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:38.021118 (+ 304us) server_negotiation.cc:407] Connection header received
0618 23:31:38.021715 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:38.021716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:38.021729 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:38.021785 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:38.022575 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.023158 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.024296 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.024518 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.024595 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.028760 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:38.028762 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:38.028762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:38.028769 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:38.030557 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:38.031042 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:38.031043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:38.031044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:38.031099 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:38.031390 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:38.031391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:38.031391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:38.031514 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:38.031604 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:38.031894 (+ 290us) server_negotiation.cc:299] Negotiation successful
0618 23:31:38.031977 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 23:31:38.032294 2502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.020708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44909 (local address 127.0.98.3:59557)
0618 23:31:38.020990 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:38.021035 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:38.021600 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:38.021863 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:38.021867 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:38.022225 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:38.022481 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.022485 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.023259 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.023260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.024196 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.024198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.024623 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.024625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.024897 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.025634 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:38.025647 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:38.028643 (+ 2996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:38.030653 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:38.030659 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:38.030662 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:38.030946 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:38.031173 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:38.031174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:38.031174 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:38.031333 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:38.031608 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:38.031614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:38.031751 (+ 137us) client_negotiation.cc:769] Sending connection context
0618 23:31:38.031873 (+ 122us) client_negotiation.cc:240] Negotiation successful
0618 23:31:38.032049 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":207,"threads_started":1}
I20260618 23:31:38.032539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 2478
I20260618 23:31:38.032637 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:31:38.032876 392 external_mini_cluster.cc:1428] Setting key d669d3a02065c564fca2ad8d7600dfeb
I20260618 23:31:38.032912 2609 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44909
I20260618 23:31:38.033036 2609 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:38.033288 2609 heartbeater.cc:511] Master 127.0.98.62:44909 requested a full tablet report, sending...
I20260618 23:31:38.033668 2126 ts_manager.cc:194] Registered new tserver with Master: faef863dbed84e0b927b5833deacec3d (127.0.98.3:45889)
I20260618 23:31:38.034114 2126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59557
I20260618 23:31:38.040500 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.034317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41901 (local address 127.0.98.62:44909)
0618 23:31:38.034357 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:38.034359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:38.034486 (+ 127us) server_negotiation.cc:407] Connection header received
0618 23:31:38.034528 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:38.034529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:38.034539 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:38.034610 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:38.034975 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.035475 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.036323 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.036505 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.036568 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.037595 (+ 1027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:38.037596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:38.037596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:38.037603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:38.039326 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:38.039728 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:38.039729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:38.039729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:38.039783 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:38.040042 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:38.040043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:38.040044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:38.040164 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:38.040235 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:38.040352 (+ 117us) server_negotiation.cc:299] Negotiation successful
0618 23:31:38.040435 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260618 23:31:38.040514 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.034241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44909 (local address 127.0.98.3:41901)
0618 23:31:38.034385 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:38.034396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:38.034473 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:38.034618 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:38.034619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:38.034811 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:38.034893 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.034895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.035641 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.035641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.036216 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.036217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.036583 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.036584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.036643 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.037167 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:38.037171 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:38.037497 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:38.039413 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:38.039414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:38.039415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:38.039648 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:38.039888 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:38.039889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:38.039889 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:38.039945 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:38.040235 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:38.040236 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:38.040278 (+ 42us) client_negotiation.cc:769] Sending connection context
0618 23:31:38.040334 (+ 56us) client_negotiation.cc:240] Negotiation successful
0618 23:31:38.040427 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260618 23:31:38.045104 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 23:31:38 dist-test-slave-6224 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 23:31:38 dist-test-slave-6224 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:38.071154 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.063416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57298 (local address 127.0.98.62:44909)
0618 23:31:38.063492 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:38.063494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:38.063507 (+ 13us) server_negotiation.cc:407] Connection header received
0618 23:31:38.063580 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:38.063582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:38.063594 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:38.063679 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:38.064296 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.064804 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.065560 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.065744 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.065799 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.068556 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:38.068557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:38.068558 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:38.068566 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:38.070108 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:38.070454 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:38.070455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:38.070455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:38.070511 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:38.070695 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:38.070695 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:38.070696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:38.070813 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:38.070895 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:38.070998 (+ 103us) server_negotiation.cc:299] Negotiation successful
0618 23:31:38.071059 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 23:31:38.073247 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:31:38.075966 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:31:38.084849 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.079256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45889 (local address 127.0.0.1:53020)
0618 23:31:38.079799 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:38.079849 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:38.080050 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:38.080385 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:38.080389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:38.080393 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:38.080507 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.080509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.081820 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.081821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.083214 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.083217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.084483 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.084484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.084586 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.084601 (+ 15us) client_negotiation.cc:769] Sending connection context
0618 23:31:38.084673 (+ 72us) client_negotiation.cc:240] Negotiation successful
0618 23:31:38.084740 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":86,"threads_started":1}
I20260618 23:31:38.085201 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.079427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53020 (local address 127.0.98.3:45889)
0618 23:31:38.079725 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:38.079731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:38.079965 (+ 234us) server_negotiation.cc:407] Connection header received
0618 23:31:38.080201 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:38.080205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:38.080227 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:38.080357 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:38.080610 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.081705 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.083347 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.084282 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.084356 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.084442 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:38.085003 (+ 561us) server_negotiation.cc:299] Negotiation successful
0618 23:31:38.085068 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":105,"threads_started":1}
I20260618 23:31:38.086169 2543 tablet_service.cc:1467] Processing CreateTablet for tablet a34d6537d9364593a8698ec138fd9d22 (DEFAULT_TABLE table=test-table [id=aeab2a617f0844edb8fd6838496934dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:38.086524 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.079812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38124 (local address 127.0.98.2:39127)
0618 23:31:38.080020 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:38.080025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:38.081850 (+ 1825us) server_negotiation.cc:407] Connection header received
0618 23:31:38.081897 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:38.081899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:38.081914 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:38.081990 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:38.082188 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.083236 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.085575 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.086085 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.086195 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.086257 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:38.086377 (+ 120us) server_negotiation.cc:299] Negotiation successful
0618 23:31:38.086422 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":96,"threads_started":1}
I20260618 23:31:38.086648 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a34d6537d9364593a8698ec138fd9d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:38.086855 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.079493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39127 (local address 127.0.0.1:38124)
0618 23:31:38.081675 (+ 2182us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:38.081687 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:38.081745 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:38.082057 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:38.082058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:38.082061 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:38.082113 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.082115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.083344 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.083346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.085470 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.085473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.086219 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.086220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.086298 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.086303 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:31:38.086363 (+ 60us) client_negotiation.cc:240] Negotiation successful
0618 23:31:38.086546 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2168,"thread_start_us":118,"threads_started":1}
I20260618 23:31:38.087918 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.081536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42811 (local address 127.0.0.1:36794)
0618 23:31:38.082053 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:38.082066 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:38.082124 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:38.083611 (+ 1487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:38.083612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:38.083614 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:38.083662 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.083664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.084963 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.084964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.086443 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.086446 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.087666 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.087667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.087738 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.087743 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:31:38.087790 (+ 47us) client_negotiation.cc:240] Negotiation successful
0618 23:31:38.087847 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":326,"thread_start_us":82,"threads_started":1}
I20260618 23:31:38.088593 2402 tablet_service.cc:1467] Processing CreateTablet for tablet a34d6537d9364593a8698ec138fd9d22 (DEFAULT_TABLE table=test-table [id=aeab2a617f0844edb8fd6838496934dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:38.089056 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a34d6537d9364593a8698ec138fd9d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:38.091082 2632 tablet_bootstrap.cc:492] T a34d6537d9364593a8698ec138fd9d22 P faef863dbed84e0b927b5833deacec3d: Bootstrap starting.
I20260618 23:31:38.091734 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.081670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36794 (local address 127.0.98.1:42811)
0618 23:31:38.083410 (+ 1740us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:38.083414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:38.083423 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:31:38.083454 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:38.083455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:38.083467 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:38.083610 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:38.083756 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.084851 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.086569 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.087539 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.087659 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.087813 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:38.087854 (+ 41us) server_negotiation.cc:299] Negotiation successful
0618 23:31:38.087906 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1694,"thread_start_us":96,"threads_started":1}
I20260618 23:31:38.092175 2632 tablet_bootstrap.cc:654] T a34d6537d9364593a8698ec138fd9d22 P faef863dbed84e0b927b5833deacec3d: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:38.092512 2632 log.cc:824] T a34d6537d9364593a8698ec138fd9d22 P faef863dbed84e0b927b5833deacec3d: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:38.092525 2263 tablet_service.cc:1467] Processing CreateTablet for tablet a34d6537d9364593a8698ec138fd9d22 (DEFAULT_TABLE table=test-table [id=aeab2a617f0844edb8fd6838496934dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:38.092888 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a34d6537d9364593a8698ec138fd9d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:38.093247 2634 tablet_bootstrap.cc:492] T a34d6537d9364593a8698ec138fd9d22 P 21e9603236b94e92b1695c9a1ed5c89d: Bootstrap starting.
I20260618 23:31:38.093971 2634 tablet_bootstrap.cc:654] T a34d6537d9364593a8698ec138fd9d22 P 21e9603236b94e92b1695c9a1ed5c89d: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:38.094362 2634 log.cc:824] T a34d6537d9364593a8698ec138fd9d22 P 21e9603236b94e92b1695c9a1ed5c89d: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:38.094383 2632 tablet_bootstrap.cc:492] T a34d6537d9364593a8698ec138fd9d22 P faef863dbed84e0b927b5833deacec3d: No bootstrap required, opened a new log
I20260618 23:31:38.094450 2632 ts_tablet_manager.cc:1397] T a34d6537d9364593a8698ec138fd9d22 P faef863dbed84e0b927b5833deacec3d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 23:31:38.095697 2634 tablet_bootstrap.cc:492] T a34d6537d9364593a8698ec138fd9d22 P 21e9603236b94e92b1695c9a1ed5c89d: No bootstrap required, opened a new log
I20260618 23:31:38.095778 2634 ts_tablet_manager.cc:1397] T a34d6537d9364593a8698ec138fd9d22 P 21e9603236b94e92b1695c9a1ed5c89d: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260618 23:31:38.096525 2632 raft_consensus.cc:348] T a34d6537d9364593a8698ec138fd9d22 P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } }
I20260618 23:31:38.096715 2632 raft_consensus.cc:374] T a34d6537d9364593a8698ec138fd9d22 P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:38.096781 2632 raft_consensus.cc:729] T a34d6537d9364593a8698ec138fd9d22 P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faef863dbed84e0b927b5833deacec3d, State: Initialized, Role: FOLLOWER
I20260618 23:31:38.096909 2632 consensus_queue.cc:260] T a34d6537d9364593a8698ec138fd9d22 P faef863dbed84e0b927b5833deacec3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } }
I20260618 23:31:38.097265 2632 ts_tablet_manager.cc:1428] T a34d6537d9364593a8698ec138fd9d22 P faef863dbed84e0b927b5833deacec3d: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260618 23:31:38.097307 2609 heartbeater.cc:503] Master 127.0.98.62:44909 was elected leader, sending a full tablet report...
I20260618 23:31:38.098033 2634 raft_consensus.cc:348] T a34d6537d9364593a8698ec138fd9d22 P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } }
I20260618 23:31:38.098210 2634 raft_consensus.cc:374] T a34d6537d9364593a8698ec138fd9d22 P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:38.098253 2634 raft_consensus.cc:729] T a34d6537d9364593a8698ec138fd9d22 P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21e9603236b94e92b1695c9a1ed5c89d, State: Initialized, Role: FOLLOWER
I20260618 23:31:38.098392 2634 consensus_queue.cc:260] T a34d6537d9364593a8698ec138fd9d22 P 21e9603236b94e92b1695c9a1ed5c89d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } }
I20260618 23:31:38.098672 2634 ts_tablet_manager.cc:1428] T a34d6537d9364593a8698ec138fd9d22 P 21e9603236b94e92b1695c9a1ed5c89d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260618 23:31:38.098765 2469 heartbeater.cc:503] Master 127.0.98.62:44909 was elected leader, sending a full tablet report...
I20260618 23:31:38.098884 2636 tablet_bootstrap.cc:492] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b: Bootstrap starting.
I20260618 23:31:38.099506 2636 tablet_bootstrap.cc:654] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:38.099920 2636 log.cc:824] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:38.100690 2636 tablet_bootstrap.cc:492] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b: No bootstrap required, opened a new log
I20260618 23:31:38.100747 2636 ts_tablet_manager.cc:1397] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260618 23:31:38.102105 2636 raft_consensus.cc:348] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } }
I20260618 23:31:38.102236 2636 raft_consensus.cc:374] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:38.102257 2636 raft_consensus.cc:729] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05450364e9e4578a1fa7034f7c23b2b, State: Initialized, Role: FOLLOWER
I20260618 23:31:38.102414 2636 consensus_queue.cc:260] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } }
I20260618 23:31:38.102697 2636 ts_tablet_manager.cc:1428] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260618 23:31:38.102787 2329 heartbeater.cc:503] Master 127.0.98.62:44909 was elected leader, sending a full tablet report...
W20260618 23:31:38.103035 2470 tablet.cc:2369] T a34d6537d9364593a8698ec138fd9d22 P 21e9603236b94e92b1695c9a1ed5c89d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:31:38.108755 2640 raft_consensus.cc:482] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:31:38.108909 2640 raft_consensus.cc:504] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } }
I20260618 23:31:38.109304 2640 leader_election.cc:288] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers faef863dbed84e0b927b5833deacec3d (127.0.98.3:45889), 21e9603236b94e92b1695c9a1ed5c89d (127.0.98.2:39127)
I20260618 23:31:38.112840 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.109693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52497 (local address 127.0.98.3:45889)
0618 23:31:38.109880 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:38.109883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:38.109896 (+ 13us) server_negotiation.cc:407] Connection header received
0618 23:31:38.109936 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:38.109937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:38.109955 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:38.110040 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:38.110251 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.110885 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.112056 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.112438 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.112532 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.112545 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:38.112709 (+ 164us) server_negotiation.cc:299] Negotiation successful
0618 23:31:38.112751 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260618 23:31:38.112840 2222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.109613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45889 (local address 127.0.98.1:52497)
0618 23:31:38.109690 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:38.109708 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:38.109858 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:38.110076 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:38.110077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:38.110079 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:38.110163 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.110166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.110981 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.110983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.111964 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.111967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.112531 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.112532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.112613 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.112617 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:31:38.112701 (+ 84us) client_negotiation.cc:240] Negotiation successful
0618 23:31:38.112751 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":4}
I20260618 23:31:38.113494 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.109761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39127 (local address 127.0.98.1:33455)
0618 23:31:38.110001 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:38.110013 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:38.110079 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:38.110251 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:38.110252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:38.110254 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:38.110336 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.110338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.111145 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.111146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.112291 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.112294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.113191 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.113193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.113280 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.113285 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:31:38.113373 (+ 88us) client_negotiation.cc:240] Negotiation successful
0618 23:31:38.113416 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":151,"threads_started":1}
I20260618 23:31:38.113533 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34d6537d9364593a8698ec138fd9d22" candidate_uuid: "c05450364e9e4578a1fa7034f7c23b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faef863dbed84e0b927b5833deacec3d" is_pre_election: true
I20260618 23:31:38.113785 2563 raft_consensus.cc:2454] T a34d6537d9364593a8698ec138fd9d22 P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05450364e9e4578a1fa7034f7c23b2b in term 0.
I20260618 23:31:38.114122 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.109888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33455 (local address 127.0.98.2:39127)
0618 23:31:38.109939 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:38.109941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:38.110087 (+ 146us) server_negotiation.cc:407] Connection header received
0618 23:31:38.110130 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:38.110130 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:38.110143 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:38.110217 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:38.110445 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.111039 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.112550 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.113078 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.113172 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.113185 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:38.113953 (+ 768us) server_negotiation.cc:299] Negotiation successful
0618 23:31:38.114062 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260618 23:31:38.114109 2217 leader_election.cc:302] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05450364e9e4578a1fa7034f7c23b2b, faef863dbed84e0b927b5833deacec3d; no voters:
I20260618 23:31:38.114295 2640 raft_consensus.cc:2781] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:31:38.114382 2640 raft_consensus.cc:482] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:31:38.114410 2640 raft_consensus.cc:3037] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:38.114429 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34d6537d9364593a8698ec138fd9d22" candidate_uuid: "c05450364e9e4578a1fa7034f7c23b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21e9603236b94e92b1695c9a1ed5c89d" is_pre_election: true
I20260618 23:31:38.114598 2423 raft_consensus.cc:2454] T a34d6537d9364593a8698ec138fd9d22 P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05450364e9e4578a1fa7034f7c23b2b in term 0.
I20260618 23:31:38.115378 2640 raft_consensus.cc:504] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } }
I20260618 23:31:38.115545 2640 leader_election.cc:288] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b [CANDIDATE]: Term 1 election: Requested vote from peers faef863dbed84e0b927b5833deacec3d (127.0.98.3:45889), 21e9603236b94e92b1695c9a1ed5c89d (127.0.98.2:39127)
I20260618 23:31:38.115813 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34d6537d9364593a8698ec138fd9d22" candidate_uuid: "c05450364e9e4578a1fa7034f7c23b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21e9603236b94e92b1695c9a1ed5c89d"
I20260618 23:31:38.115890 2423 raft_consensus.cc:3037] T a34d6537d9364593a8698ec138fd9d22 P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:38.116667 2423 raft_consensus.cc:2454] T a34d6537d9364593a8698ec138fd9d22 P 21e9603236b94e92b1695c9a1ed5c89d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05450364e9e4578a1fa7034f7c23b2b in term 1.
I20260618 23:31:38.116797 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a34d6537d9364593a8698ec138fd9d22" candidate_uuid: "c05450364e9e4578a1fa7034f7c23b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faef863dbed84e0b927b5833deacec3d"
I20260618 23:31:38.116916 2217 leader_election.cc:302] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21e9603236b94e92b1695c9a1ed5c89d, c05450364e9e4578a1fa7034f7c23b2b; no voters:
I20260618 23:31:38.116914 2563 raft_consensus.cc:3037] T a34d6537d9364593a8698ec138fd9d22 P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:38.117063 2640 raft_consensus.cc:2781] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:38.117339 2640 raft_consensus.cc:686] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b [term 1 LEADER]: Becoming Leader. State: Replica: c05450364e9e4578a1fa7034f7c23b2b, State: Running, Role: LEADER
I20260618 23:31:38.117457 2640 consensus_queue.cc:237] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } }
I20260618 23:31:38.117782 2563 raft_consensus.cc:2454] T a34d6537d9364593a8698ec138fd9d22 P faef863dbed84e0b927b5833deacec3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05450364e9e4578a1fa7034f7c23b2b in term 1.
I20260618 23:31:38.118462 2125 catalog_manager.cc:5495] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b reported cstate change: term changed from 0 to 1, leader changed from <none> to c05450364e9e4578a1fa7034f7c23b2b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c05450364e9e4578a1fa7034f7c23b2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } health_report { overall_health: UNKNOWN } } }
Jun 18 23:31:38 dist-test-slave-6224 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825497, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:38.172919 2175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.164438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57314 (local address 127.0.98.62:44909)
0618 23:31:38.164545 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:38.164548 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:38.164686 (+ 138us) server_negotiation.cc:407] Connection header received
0618 23:31:38.164799 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:38.164800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:38.164814 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:38.164888 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:38.165495 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.166018 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.166787 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.167011 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.167095 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.169786 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:38.169788 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:38.169788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:38.169795 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:38.171401 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:38.171955 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:38.171956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:38.171957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:38.172015 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:38.172286 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:38.172287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:38.172287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:38.172412 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:38.172473 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:38.172663 (+ 190us) server_negotiation.cc:299] Negotiation successful
0618 23:31:38.172797 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":50}
I20260618 23:31:38.172923 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.164332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44909 (local address 127.0.0.1:57314)
0618 23:31:38.164570 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:38.164584 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:38.164718 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:38.164892 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:38.164893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:38.165335 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:38.165393 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.165396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.166109 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.166110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.166693 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.166695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.167095 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.167096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.167144 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.167894 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:38.167900 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:38.169668 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:38.171514 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:38.171522 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:38.171524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:38.171855 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:38.172123 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:38.172124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:38.172125 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:38.172199 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:38.172489 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:38.172497 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:38.172557 (+ 60us) client_negotiation.cc:769] Sending connection context
0618 23:31:38.172627 (+ 70us) client_negotiation.cc:240] Negotiation successful
0618 23:31:38.172747 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":111,"threads_started":1}
I20260618 23:31:38.175722 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57314:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260618 23:31:38.175938 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:31:38.177855 2263 tablet_service.cc:1467] Processing CreateTablet for tablet 9a6b53451f47425c8cfda6c52ea3eaf1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ea04f47eb624433fb959df292c23bb53]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 23:31:38.177855 2402 tablet_service.cc:1467] Processing CreateTablet for tablet 9a6b53451f47425c8cfda6c52ea3eaf1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ea04f47eb624433fb959df292c23bb53]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 23:31:38.177913 2543 tablet_service.cc:1467] Processing CreateTablet for tablet 9a6b53451f47425c8cfda6c52ea3eaf1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ea04f47eb624433fb959df292c23bb53]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 23:31:38.178040 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a6b53451f47425c8cfda6c52ea3eaf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:38.178040 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a6b53451f47425c8cfda6c52ea3eaf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:38.178169 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a6b53451f47425c8cfda6c52ea3eaf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:38.179540 2632 tablet_bootstrap.cc:492] T 9a6b53451f47425c8cfda6c52ea3eaf1 P faef863dbed84e0b927b5833deacec3d: Bootstrap starting.
I20260618 23:31:38.179760 2634 tablet_bootstrap.cc:492] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d: Bootstrap starting.
I20260618 23:31:38.179769 2636 tablet_bootstrap.cc:492] T 9a6b53451f47425c8cfda6c52ea3eaf1 P c05450364e9e4578a1fa7034f7c23b2b: Bootstrap starting.
I20260618 23:31:38.180205 2632 tablet_bootstrap.cc:654] T 9a6b53451f47425c8cfda6c52ea3eaf1 P faef863dbed84e0b927b5833deacec3d: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:38.180512 2636 tablet_bootstrap.cc:654] T 9a6b53451f47425c8cfda6c52ea3eaf1 P c05450364e9e4578a1fa7034f7c23b2b: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:38.180512 2634 tablet_bootstrap.cc:654] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:38.180953 2632 tablet_bootstrap.cc:492] T 9a6b53451f47425c8cfda6c52ea3eaf1 P faef863dbed84e0b927b5833deacec3d: No bootstrap required, opened a new log
I20260618 23:31:38.181010 2632 ts_tablet_manager.cc:1397] T 9a6b53451f47425c8cfda6c52ea3eaf1 P faef863dbed84e0b927b5833deacec3d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:38.181193 2632 raft_consensus.cc:348] T 9a6b53451f47425c8cfda6c52ea3eaf1 P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:38.181290 2632 raft_consensus.cc:374] T 9a6b53451f47425c8cfda6c52ea3eaf1 P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:38.181295 2634 tablet_bootstrap.cc:492] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d: No bootstrap required, opened a new log
I20260618 23:31:38.181317 2632 raft_consensus.cc:729] T 9a6b53451f47425c8cfda6c52ea3eaf1 P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faef863dbed84e0b927b5833deacec3d, State: Initialized, Role: FOLLOWER
I20260618 23:31:38.181332 2634 ts_tablet_manager.cc:1397] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260618 23:31:38.181372 2632 consensus_queue.cc:260] T 9a6b53451f47425c8cfda6c52ea3eaf1 P faef863dbed84e0b927b5833deacec3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:38.181475 2632 ts_tablet_manager.cc:1428] T 9a6b53451f47425c8cfda6c52ea3eaf1 P faef863dbed84e0b927b5833deacec3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:38.181545 2636 tablet_bootstrap.cc:492] T 9a6b53451f47425c8cfda6c52ea3eaf1 P c05450364e9e4578a1fa7034f7c23b2b: No bootstrap required, opened a new log
I20260618 23:31:38.181519 2634 raft_consensus.cc:348] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:38.181542 2637 tablet_replica.cc:439] T 9a6b53451f47425c8cfda6c52ea3eaf1 P faef863dbed84e0b927b5833deacec3d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } }
I20260618 23:31:38.181603 2636 ts_tablet_manager.cc:1397] T 9a6b53451f47425c8cfda6c52ea3eaf1 P c05450364e9e4578a1fa7034f7c23b2b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260618 23:31:38.181607 2634 raft_consensus.cc:374] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:38.181631 2634 raft_consensus.cc:729] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21e9603236b94e92b1695c9a1ed5c89d, State: Initialized, Role: FOLLOWER
I20260618 23:31:38.181627 2637 tablet_replica.cc:442] T 9a6b53451f47425c8cfda6c52ea3eaf1 P faef863dbed84e0b927b5833deacec3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:31:38.181692 2634 consensus_queue.cc:260] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:38.181749 2636 raft_consensus.cc:348] T 9a6b53451f47425c8cfda6c52ea3eaf1 P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:38.181787 2634 ts_tablet_manager.cc:1428] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260618 23:31:38.181810 2636 raft_consensus.cc:374] T 9a6b53451f47425c8cfda6c52ea3eaf1 P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:38.181835 2636 raft_consensus.cc:729] T 9a6b53451f47425c8cfda6c52ea3eaf1 P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05450364e9e4578a1fa7034f7c23b2b, State: Initialized, Role: FOLLOWER
I20260618 23:31:38.181891 2636 consensus_queue.cc:260] T 9a6b53451f47425c8cfda6c52ea3eaf1 P c05450364e9e4578a1fa7034f7c23b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:38.181983 2636 ts_tablet_manager.cc:1428] T 9a6b53451f47425c8cfda6c52ea3eaf1 P c05450364e9e4578a1fa7034f7c23b2b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260618 23:31:38.181933 2638 tablet_replica.cc:439] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } }
I20260618 23:31:38.182032 2638 tablet_replica.cc:442] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:31:38.182142 2640 tablet_replica.cc:439] T 9a6b53451f47425c8cfda6c52ea3eaf1 P c05450364e9e4578a1fa7034f7c23b2b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } }
I20260618 23:31:38.182292 2640 tablet_replica.cc:442] T 9a6b53451f47425c8cfda6c52ea3eaf1 P c05450364e9e4578a1fa7034f7c23b2b: This TxnStatusTablet replica's current role is: FOLLOWER
W20260618 23:31:38.191233 2330 tablet.cc:2369] T 9a6b53451f47425c8cfda6c52ea3eaf1 P c05450364e9e4578a1fa7034f7c23b2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:31:38.280401 2610 tablet.cc:2369] T 9a6b53451f47425c8cfda6c52ea3eaf1 P faef863dbed84e0b927b5833deacec3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:31:38.298611 2638 raft_consensus.cc:482] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:31:38.298781 2638 raft_consensus.cc:504] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:38.299386 2638 leader_election.cc:288] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c05450364e9e4578a1fa7034f7c23b2b (127.0.98.1:42811), faef863dbed84e0b927b5833deacec3d (127.0.98.3:45889)
I20260618 23:31:38.302733 2362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.299613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45889 (local address 127.0.98.2:56209)
0618 23:31:38.299697 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:38.299715 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:38.299782 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:38.299956 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:38.299957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:38.299959 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:38.300074 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.300077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.300852 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.300853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.301797 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.301799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.302421 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.302422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.302514 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.302519 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:31:38.302591 (+ 72us) client_negotiation.cc:240] Negotiation successful
0618 23:31:38.302645 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":34}
I20260618 23:31:38.302779 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.299710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56209 (local address 127.0.98.3:45889)
0618 23:31:38.299767 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:38.299770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:38.299787 (+ 17us) server_negotiation.cc:407] Connection header received
0618 23:31:38.299859 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:38.299860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:38.299871 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:38.299942 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:38.300176 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.300744 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.301903 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.302332 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.302391 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.302401 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:38.302646 (+ 245us) server_negotiation.cc:299] Negotiation successful
0618 23:31:38.302697 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260618 23:31:38.303056 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.299717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39231 (local address 127.0.98.1:42811)
0618 23:31:38.299767 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:38.299770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:38.300053 (+ 283us) server_negotiation.cc:407] Connection header received
0618 23:31:38.300090 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:38.300091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:38.300106 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:38.300174 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:38.300405 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.301011 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.302074 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.302517 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.302599 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.302611 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:38.302906 (+ 295us) server_negotiation.cc:299] Negotiation successful
0618 23:31:38.302989 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260618 23:31:38.303077 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6b53451f47425c8cfda6c52ea3eaf1" candidate_uuid: "21e9603236b94e92b1695c9a1ed5c89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faef863dbed84e0b927b5833deacec3d" is_pre_election: true
I20260618 23:31:38.303198 2563 raft_consensus.cc:2454] T 9a6b53451f47425c8cfda6c52ea3eaf1 P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21e9603236b94e92b1695c9a1ed5c89d in term 0.
I20260618 23:31:38.303380 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.299613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42811 (local address 127.0.98.2:39231)
0618 23:31:38.299944 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:38.299959 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:38.300033 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:38.300197 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:38.300198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:38.300200 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:38.300279 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.300281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.301117 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.301118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.301979 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.301982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.302671 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.302672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.302750 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.302756 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:31:38.303207 (+ 451us) client_negotiation.cc:240] Negotiation successful
0618 23:31:38.303265 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":65,"thread_start_us":136,"threads_started":1}
I20260618 23:31:38.303468 2357 leader_election.cc:302] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21e9603236b94e92b1695c9a1ed5c89d, faef863dbed84e0b927b5833deacec3d; no voters:
I20260618 23:31:38.303665 2638 raft_consensus.cc:2781] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:31:38.303725 2638 raft_consensus.cc:482] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:31:38.303748 2638 raft_consensus.cc:3037] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:38.303793 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6b53451f47425c8cfda6c52ea3eaf1" candidate_uuid: "21e9603236b94e92b1695c9a1ed5c89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05450364e9e4578a1fa7034f7c23b2b" is_pre_election: true
I20260618 23:31:38.303925 2283 raft_consensus.cc:2454] T 9a6b53451f47425c8cfda6c52ea3eaf1 P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21e9603236b94e92b1695c9a1ed5c89d in term 0.
I20260618 23:31:38.304555 2638 raft_consensus.cc:504] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:38.304723 2638 leader_election.cc:288] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d [CANDIDATE]: Term 1 election: Requested vote from peers c05450364e9e4578a1fa7034f7c23b2b (127.0.98.1:42811), faef863dbed84e0b927b5833deacec3d (127.0.98.3:45889)
I20260618 23:31:38.304845 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6b53451f47425c8cfda6c52ea3eaf1" candidate_uuid: "21e9603236b94e92b1695c9a1ed5c89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05450364e9e4578a1fa7034f7c23b2b"
I20260618 23:31:38.304932 2283 raft_consensus.cc:3037] T 9a6b53451f47425c8cfda6c52ea3eaf1 P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:38.304989 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a6b53451f47425c8cfda6c52ea3eaf1" candidate_uuid: "21e9603236b94e92b1695c9a1ed5c89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faef863dbed84e0b927b5833deacec3d"
I20260618 23:31:38.305080 2563 raft_consensus.cc:3037] T 9a6b53451f47425c8cfda6c52ea3eaf1 P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:38.305712 2563 raft_consensus.cc:2454] T 9a6b53451f47425c8cfda6c52ea3eaf1 P faef863dbed84e0b927b5833deacec3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21e9603236b94e92b1695c9a1ed5c89d in term 1.
I20260618 23:31:38.305747 2283 raft_consensus.cc:2454] T 9a6b53451f47425c8cfda6c52ea3eaf1 P c05450364e9e4578a1fa7034f7c23b2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21e9603236b94e92b1695c9a1ed5c89d in term 1.
I20260618 23:31:38.305904 2357 leader_election.cc:302] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21e9603236b94e92b1695c9a1ed5c89d, faef863dbed84e0b927b5833deacec3d; no voters:
I20260618 23:31:38.305997 2638 raft_consensus.cc:2781] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:38.306180 2638 raft_consensus.cc:686] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d [term 1 LEADER]: Becoming Leader. State: Replica: 21e9603236b94e92b1695c9a1ed5c89d, State: Running, Role: LEADER
I20260618 23:31:38.306315 2638 consensus_queue.cc:237] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:38.306763 2654 tablet_replica.cc:439] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d: TxnStatusTablet state changed. Reason: New leader 21e9603236b94e92b1695c9a1ed5c89d. Latest consensus state: current_term: 1 leader_uuid: "21e9603236b94e92b1695c9a1ed5c89d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } }
I20260618 23:31:38.306938 2654 tablet_replica.cc:442] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d: This TxnStatusTablet replica's current role is: LEADER
I20260618 23:31:38.307166 2125 catalog_manager.cc:5495] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d reported cstate change: term changed from 0 to 1, leader changed from <none> to 21e9603236b94e92b1695c9a1ed5c89d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "21e9603236b94e92b1695c9a1ed5c89d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } health_report { overall_health: UNKNOWN } } }
I20260618 23:31:38.307498 2656 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260618 23:31:38.307662 2656 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260618 23:31:38.335253 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.331635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38136 (local address 127.0.98.2:39127)
0618 23:31:38.331701 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:38.331703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:38.331719 (+ 16us) server_negotiation.cc:407] Connection header received
0618 23:31:38.331779 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:38.331780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:38.331798 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:38.331910 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:31:38.332101 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.332860 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.334164 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.334533 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.334613 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.334861 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:31:38.334985 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:31:38.335054 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:38.335140 (+ 86us) server_negotiation.cc:299] Negotiation successful
0618 23:31:38.335169 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260618 23:31:38.335305 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.331463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39127 (local address 127.0.0.1:38136)
0618 23:31:38.331600 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:38.331615 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:38.331670 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:38.331943 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:38.331944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:38.331946 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:31:38.332021 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.332025 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.332951 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.332952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.334071 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.334074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.334634 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.334635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.334718 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.334743 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:31:38.335060 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:31:38.335068 (+ 8us) client_negotiation.cc:769] Sending connection context
0618 23:31:38.335132 (+ 64us) client_negotiation.cc:240] Negotiation successful
0618 23:31:38.335199 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260618 23:31:38.337333 2563 raft_consensus.cc:1261] T 9a6b53451f47425c8cfda6c52ea3eaf1 P faef863dbed84e0b927b5833deacec3d [term 1 FOLLOWER]: Refusing update from remote peer 21e9603236b94e92b1695c9a1ed5c89d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:31:38.337517 2283 raft_consensus.cc:1261] T 9a6b53451f47425c8cfda6c52ea3eaf1 P c05450364e9e4578a1fa7034f7c23b2b [term 1 FOLLOWER]: Refusing update from remote peer 21e9603236b94e92b1695c9a1ed5c89d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:31:38.337792 2654 consensus_queue.cc:1035] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d [LEADER]: Connected to new peer: Peer: permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:31:38.337963 2638 consensus_queue.cc:1035] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:31:38.339471 2640 tablet_replica.cc:439] T 9a6b53451f47425c8cfda6c52ea3eaf1 P c05450364e9e4578a1fa7034f7c23b2b: TxnStatusTablet state changed. Reason: New leader 21e9603236b94e92b1695c9a1ed5c89d. Latest consensus state: current_term: 1 leader_uuid: "21e9603236b94e92b1695c9a1ed5c89d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } }
I20260618 23:31:38.339617 2640 tablet_replica.cc:442] T 9a6b53451f47425c8cfda6c52ea3eaf1 P c05450364e9e4578a1fa7034f7c23b2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:31:38.339838 2637 tablet_replica.cc:439] T 9a6b53451f47425c8cfda6c52ea3eaf1 P faef863dbed84e0b927b5833deacec3d: TxnStatusTablet state changed. Reason: New leader 21e9603236b94e92b1695c9a1ed5c89d. Latest consensus state: current_term: 1 leader_uuid: "21e9603236b94e92b1695c9a1ed5c89d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } }
I20260618 23:31:38.339955 2637 tablet_replica.cc:442] T 9a6b53451f47425c8cfda6c52ea3eaf1 P faef863dbed84e0b927b5833deacec3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:31:38.341498 2637 tablet_replica.cc:439] T 9a6b53451f47425c8cfda6c52ea3eaf1 P faef863dbed84e0b927b5833deacec3d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21e9603236b94e92b1695c9a1ed5c89d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } }
I20260618 23:31:38.341590 2637 tablet_replica.cc:442] T 9a6b53451f47425c8cfda6c52ea3eaf1 P faef863dbed84e0b927b5833deacec3d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:31:38.341612 2638 tablet_replica.cc:439] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21e9603236b94e92b1695c9a1ed5c89d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } }
I20260618 23:31:38.341607 2640 tablet_replica.cc:439] T 9a6b53451f47425c8cfda6c52ea3eaf1 P c05450364e9e4578a1fa7034f7c23b2b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "21e9603236b94e92b1695c9a1ed5c89d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } }
I20260618 23:31:38.341710 2638 tablet_replica.cc:442] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d: This TxnStatusTablet replica's current role is: LEADER
I20260618 23:31:38.341722 2640 tablet_replica.cc:442] T 9a6b53451f47425c8cfda6c52ea3eaf1 P c05450364e9e4578a1fa7034f7c23b2b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:31:38.342070 2664 tablet_replica.cc:439] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "21e9603236b94e92b1695c9a1ed5c89d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } } }
I20260618 23:31:38.342147 2664 tablet_replica.cc:442] T 9a6b53451f47425c8cfda6c52ea3eaf1 P 21e9603236b94e92b1695c9a1ed5c89d: This TxnStatusTablet replica's current role is: LEADER
I20260618 23:31:38.342271 2659 mvcc.cc:205] Tried to move back new op lower bound from 7298357241186414592 to 7298357241063174144. Current Snapshot: MvccSnapshot[applied={T|T < 7298357241186414592}]
I20260618 23:31:38.342473 2660 mvcc.cc:205] Tried to move back new op lower bound from 7298357241186414592 to 7298357241063174144. Current Snapshot: MvccSnapshot[applied={T|T < 7298357241186414592}]
I20260618 23:31:38.342859 2661 mvcc.cc:205] Tried to move back new op lower bound from 7298357241186414592 to 7298357241063174144. Current Snapshot: MvccSnapshot[applied={T|T < 7298357241186414592}]
I20260618 23:31:38.347712 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.344753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36808 (local address 127.0.98.1:42811)
0618 23:31:38.344801 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:38.344803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:38.344858 (+ 55us) server_negotiation.cc:407] Connection header received
0618 23:31:38.344974 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:38.344975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:38.344992 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:38.345060 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:31:38.345240 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.345727 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.346922 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.347108 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.347163 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.347348 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:31:38.347451 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:31:38.347514 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:38.347623 (+ 109us) server_negotiation.cc:299] Negotiation successful
0618 23:31:38.347652 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:31:38.353219 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.350616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45005 (local address 127.0.98.2:39127)
0618 23:31:38.350657 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:38.350660 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:38.350806 (+ 146us) server_negotiation.cc:407] Connection header received
0618 23:31:38.350876 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:38.350877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:38.350887 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:38.350974 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:31:38.351096 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.351648 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.352504 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.352702 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.352773 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.352969 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:31:38.353001 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:31:38.353065 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:38.353125 (+ 60us) server_negotiation.cc:299] Negotiation successful
0618 23:31:38.353147 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260618 23:31:38.353250 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.350518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39127 (local address 127.0.98.1:45005)
0618 23:31:38.350723 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:38.350744 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:38.350824 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:38.350980 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:38.350981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:38.350982 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:31:38.351046 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.351047 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.351725 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.351726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.352397 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.352399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.352783 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.352784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.352841 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.352861 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:31:38.353080 (+ 219us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:31:38.353082 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:31:38.353124 (+ 42us) client_negotiation.cc:240] Negotiation successful
0618 23:31:38.353162 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":120,"threads_started":1}
I20260618 23:31:38.356031 2423 raft_consensus.cc:1261] T a34d6537d9364593a8698ec138fd9d22 P 21e9603236b94e92b1695c9a1ed5c89d [term 1 FOLLOWER]: Refusing update from remote peer c05450364e9e4578a1fa7034f7c23b2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:31:38.356274 2563 raft_consensus.cc:1261] T a34d6537d9364593a8698ec138fd9d22 P faef863dbed84e0b927b5833deacec3d [term 1 FOLLOWER]: Refusing update from remote peer c05450364e9e4578a1fa7034f7c23b2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:31:38.356464 2640 consensus_queue.cc:1035] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:31:38.356686 2642 consensus_queue.cc:1035] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:31:38.367065 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.364593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47385 (local address 127.0.98.1:42811)
0618 23:31:38.364646 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:38.364649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:38.364658 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:31:38.364688 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:38.364689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:38.364704 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:38.364764 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:31:38.364923 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.365416 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.366206 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.366470 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.366543 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.366808 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:31:38.366844 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:31:38.366898 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:38.366964 (+ 66us) server_negotiation.cc:299] Negotiation successful
0618 23:31:38.366987 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260618 23:31:38.367077 2476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:38.364502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42811 (local address 127.0.98.2:47385)
0618 23:31:38.364579 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:38.364592 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:38.364633 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:38.364785 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:38.364785 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:38.364787 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:31:38.364846 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.364847 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.365510 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.365511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.366099 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:38.366100 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:38.366560 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:38.366562 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:38.366666 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:38.366688 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:31:38.366902 (+ 214us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:31:38.366904 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:31:38.366959 (+ 55us) client_negotiation.cc:240] Negotiation successful
0618 23:31:38.367011 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260618 23:31:38.378930 2124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:31:38.379104 2124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:31:38.380105 2124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918: Sending DeleteTablet for 3 replicas of tablet a34d6537d9364593a8698ec138fd9d22
I20260618 23:31:38.380486 2263 tablet_service.cc:1514] Processing DeleteTablet for tablet a34d6537d9364593a8698ec138fd9d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36794
I20260618 23:31:38.380651 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet a34d6537d9364593a8698ec138fd9d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53020
I20260618 23:31:38.380906 2680 tablet_replica.cc:330] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b: stopping tablet replica
I20260618 23:31:38.381064 2680 raft_consensus.cc:2229] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:31:38.381223 2680 raft_consensus.cc:2258] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:38.381201 2124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:31:38.381398 2124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:31:38.381891 2402 tablet_service.cc:1514] Processing DeleteTablet for tablet a34d6537d9364593a8698ec138fd9d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38124
I20260618 23:31:38.382647 2680 ts_tablet_manager.cc:1905] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:31:38.383047 2263 tablet_service.cc:1467] Processing CreateTablet for tablet abc69b719e874d9ba82e38bc96d0850a (DEFAULT_TABLE table=test-table [id=a6bd2310584f49d5a1fcf7b952369958]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:38.383224 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abc69b719e874d9ba82e38bc96d0850a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:38.385022 2636 tablet_bootstrap.cc:492] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b: Bootstrap starting.
I20260618 23:31:38.385215 2680 ts_tablet_manager.cc:1918] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:31:38.385344 2680 log.cc:1196] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-0/wal/wals/a34d6537d9364593a8698ec138fd9d22
I20260618 23:31:38.385488 2636 tablet_bootstrap.cc:654] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:38.385630 2543 tablet_service.cc:1467] Processing CreateTablet for tablet abc69b719e874d9ba82e38bc96d0850a (DEFAULT_TABLE table=test-table [id=a6bd2310584f49d5a1fcf7b952369958]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:38.385797 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abc69b719e874d9ba82e38bc96d0850a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:38.385874 2680 ts_tablet_manager.cc:1939] T a34d6537d9364593a8698ec138fd9d22 P c05450364e9e4578a1fa7034f7c23b2b: Deleting consensus metadata
I20260618 23:31:38.386392 2112 catalog_manager.cc:4841] TS c05450364e9e4578a1fa7034f7c23b2b (127.0.98.1:42811): tablet a34d6537d9364593a8698ec138fd9d22 (table test-table [id=aeab2a617f0844edb8fd6838496934dd]) successfully deleted
I20260618 23:31:38.386653 2682 tablet_replica.cc:330] T a34d6537d9364593a8698ec138fd9d22 P 21e9603236b94e92b1695c9a1ed5c89d: stopping tablet replica
I20260618 23:31:38.386847 2682 raft_consensus.cc:2229] T a34d6537d9364593a8698ec138fd9d22 P 21e9603236b94e92b1695c9a1ed5c89d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:31:38.386978 2682 raft_consensus.cc:2258] T a34d6537d9364593a8698ec138fd9d22 P 21e9603236b94e92b1695c9a1ed5c89d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:38.387395 2632 tablet_bootstrap.cc:492] T abc69b719e874d9ba82e38bc96d0850a P faef863dbed84e0b927b5833deacec3d: Bootstrap starting.
I20260618 23:31:38.387880 2632 tablet_bootstrap.cc:654] T abc69b719e874d9ba82e38bc96d0850a P faef863dbed84e0b927b5833deacec3d: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:38.387897 2402 tablet_service.cc:1467] Processing CreateTablet for tablet abc69b719e874d9ba82e38bc96d0850a (DEFAULT_TABLE table=test-table [id=a6bd2310584f49d5a1fcf7b952369958]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:38.388025 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abc69b719e874d9ba82e38bc96d0850a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:38.388409 2681 tablet_replica.cc:330] T a34d6537d9364593a8698ec138fd9d22 P faef863dbed84e0b927b5833deacec3d: stopping tablet replica
I20260618 23:31:38.388538 2681 raft_consensus.cc:2229] T a34d6537d9364593a8698ec138fd9d22 P faef863dbed84e0b927b5833deacec3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:31:38.388612 2681 raft_consensus.cc:2258] T a34d6537d9364593a8698ec138fd9d22 P faef863dbed84e0b927b5833deacec3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:38.388764 2682 ts_tablet_manager.cc:1905] T a34d6537d9364593a8698ec138fd9d22 P 21e9603236b94e92b1695c9a1ed5c89d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:31:38.389810 2634 tablet_bootstrap.cc:492] T abc69b719e874d9ba82e38bc96d0850a P 21e9603236b94e92b1695c9a1ed5c89d: Bootstrap starting.
I20260618 23:31:38.390249 2634 tablet_bootstrap.cc:654] T abc69b719e874d9ba82e38bc96d0850a P 21e9603236b94e92b1695c9a1ed5c89d: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:38.390844 2682 ts_tablet_manager.cc:1918] T a34d6537d9364593a8698ec138fd9d22 P 21e9603236b94e92b1695c9a1ed5c89d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:31:38.390985 2682 log.cc:1196] T a34d6537d9364593a8698ec138fd9d22 P 21e9603236b94e92b1695c9a1ed5c89d: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/a34d6537d9364593a8698ec138fd9d22
I20260618 23:31:38.391294 2682 ts_tablet_manager.cc:1939] T a34d6537d9364593a8698ec138fd9d22 P 21e9603236b94e92b1695c9a1ed5c89d: Deleting consensus metadata
I20260618 23:31:38.391952 2113 catalog_manager.cc:4841] TS 21e9603236b94e92b1695c9a1ed5c89d (127.0.98.2:39127): tablet a34d6537d9364593a8698ec138fd9d22 (table test-table [id=aeab2a617f0844edb8fd6838496934dd]) successfully deleted
I20260618 23:31:38.392088 2632 tablet_bootstrap.cc:492] T abc69b719e874d9ba82e38bc96d0850a P faef863dbed84e0b927b5833deacec3d: No bootstrap required, opened a new log
I20260618 23:31:38.392140 2632 ts_tablet_manager.cc:1397] T abc69b719e874d9ba82e38bc96d0850a P faef863dbed84e0b927b5833deacec3d: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260618 23:31:38.392295 2632 raft_consensus.cc:348] T abc69b719e874d9ba82e38bc96d0850a P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:38.392372 2632 raft_consensus.cc:374] T abc69b719e874d9ba82e38bc96d0850a P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:38.392397 2632 raft_consensus.cc:729] T abc69b719e874d9ba82e38bc96d0850a P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faef863dbed84e0b927b5833deacec3d, State: Initialized, Role: FOLLOWER
I20260618 23:31:38.392460 2632 consensus_queue.cc:260] T abc69b719e874d9ba82e38bc96d0850a P faef863dbed84e0b927b5833deacec3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:38.392555 2632 ts_tablet_manager.cc:1428] T abc69b719e874d9ba82e38bc96d0850a P faef863dbed84e0b927b5833deacec3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:38.393249 2636 tablet_bootstrap.cc:492] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b: No bootstrap required, opened a new log
I20260618 23:31:38.393322 2636 ts_tablet_manager.cc:1397] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20260618 23:31:38.392391 2681 ts_tablet_manager.cc:1905] T a34d6537d9364593a8698ec138fd9d22 P faef863dbed84e0b927b5833deacec3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:31:38.393519 2636 raft_consensus.cc:348] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:38.393630 2636 raft_consensus.cc:374] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:38.393661 2636 raft_consensus.cc:729] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05450364e9e4578a1fa7034f7c23b2b, State: Initialized, Role: FOLLOWER
I20260618 23:31:38.393709 2636 consensus_queue.cc:260] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:38.393813 2636 ts_tablet_manager.cc:1428] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260618 23:31:38.395519 2681 ts_tablet_manager.cc:1918] T a34d6537d9364593a8698ec138fd9d22 P faef863dbed84e0b927b5833deacec3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:31:38.395664 2681 log.cc:1196] T a34d6537d9364593a8698ec138fd9d22 P faef863dbed84e0b927b5833deacec3d: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/a34d6537d9364593a8698ec138fd9d22
I20260618 23:31:38.395977 2681 ts_tablet_manager.cc:1939] T a34d6537d9364593a8698ec138fd9d22 P faef863dbed84e0b927b5833deacec3d: Deleting consensus metadata
I20260618 23:31:38.396503 2113 catalog_manager.cc:4841] TS faef863dbed84e0b927b5833deacec3d (127.0.98.3:45889): tablet a34d6537d9364593a8698ec138fd9d22 (table test-table [id=aeab2a617f0844edb8fd6838496934dd]) successfully deleted
I20260618 23:31:38.396865 2634 tablet_bootstrap.cc:492] T abc69b719e874d9ba82e38bc96d0850a P 21e9603236b94e92b1695c9a1ed5c89d: No bootstrap required, opened a new log
I20260618 23:31:38.396934 2634 ts_tablet_manager.cc:1397] T abc69b719e874d9ba82e38bc96d0850a P 21e9603236b94e92b1695c9a1ed5c89d: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260618 23:31:38.397133 2634 raft_consensus.cc:348] T abc69b719e874d9ba82e38bc96d0850a P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:38.397238 2634 raft_consensus.cc:374] T abc69b719e874d9ba82e38bc96d0850a P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:38.397260 2634 raft_consensus.cc:729] T abc69b719e874d9ba82e38bc96d0850a P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21e9603236b94e92b1695c9a1ed5c89d, State: Initialized, Role: FOLLOWER
I20260618 23:31:38.397310 2634 consensus_queue.cc:260] T abc69b719e874d9ba82e38bc96d0850a P 21e9603236b94e92b1695c9a1ed5c89d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:38.397405 2634 ts_tablet_manager.cc:1428] T abc69b719e874d9ba82e38bc96d0850a P 21e9603236b94e92b1695c9a1ed5c89d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260618 23:31:38.397665 2676 raft_consensus.cc:482] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:31:38.397750 2676 raft_consensus.cc:504] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:38.397929 2676 leader_election.cc:288] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21e9603236b94e92b1695c9a1ed5c89d (127.0.98.2:39127), faef863dbed84e0b927b5833deacec3d (127.0.98.3:45889)
I20260618 23:31:38.398200 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc69b719e874d9ba82e38bc96d0850a" candidate_uuid: "c05450364e9e4578a1fa7034f7c23b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faef863dbed84e0b927b5833deacec3d" is_pre_election: true
I20260618 23:31:38.398316 2563 raft_consensus.cc:2454] T abc69b719e874d9ba82e38bc96d0850a P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05450364e9e4578a1fa7034f7c23b2b in term 0.
I20260618 23:31:38.398512 2217 leader_election.cc:302] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05450364e9e4578a1fa7034f7c23b2b, faef863dbed84e0b927b5833deacec3d; no voters:
I20260618 23:31:38.398680 2676 raft_consensus.cc:2781] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:31:38.398732 2676 raft_consensus.cc:482] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:31:38.398754 2676 raft_consensus.cc:3037] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:38.398821 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc69b719e874d9ba82e38bc96d0850a" candidate_uuid: "c05450364e9e4578a1fa7034f7c23b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21e9603236b94e92b1695c9a1ed5c89d" is_pre_election: true
I20260618 23:31:38.398900 2423 raft_consensus.cc:2454] T abc69b719e874d9ba82e38bc96d0850a P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05450364e9e4578a1fa7034f7c23b2b in term 0.
I20260618 23:31:38.399796 2676 raft_consensus.cc:504] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:38.399950 2676 leader_election.cc:288] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b [CANDIDATE]: Term 1 election: Requested vote from peers 21e9603236b94e92b1695c9a1ed5c89d (127.0.98.2:39127), faef863dbed84e0b927b5833deacec3d (127.0.98.3:45889)
I20260618 23:31:38.400192 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc69b719e874d9ba82e38bc96d0850a" candidate_uuid: "c05450364e9e4578a1fa7034f7c23b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faef863dbed84e0b927b5833deacec3d"
I20260618 23:31:38.400225 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abc69b719e874d9ba82e38bc96d0850a" candidate_uuid: "c05450364e9e4578a1fa7034f7c23b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21e9603236b94e92b1695c9a1ed5c89d"
I20260618 23:31:38.400277 2563 raft_consensus.cc:3037] T abc69b719e874d9ba82e38bc96d0850a P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:38.400298 2423 raft_consensus.cc:3037] T abc69b719e874d9ba82e38bc96d0850a P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:38.401011 2563 raft_consensus.cc:2454] T abc69b719e874d9ba82e38bc96d0850a P faef863dbed84e0b927b5833deacec3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05450364e9e4578a1fa7034f7c23b2b in term 1.
I20260618 23:31:38.401304 2423 raft_consensus.cc:2454] T abc69b719e874d9ba82e38bc96d0850a P 21e9603236b94e92b1695c9a1ed5c89d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05450364e9e4578a1fa7034f7c23b2b in term 1.
I20260618 23:31:38.401510 2217 leader_election.cc:302] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c05450364e9e4578a1fa7034f7c23b2b, faef863dbed84e0b927b5833deacec3d; no voters:
I20260618 23:31:38.401669 2676 raft_consensus.cc:2781] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:38.401727 2676 raft_consensus.cc:686] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b [term 1 LEADER]: Becoming Leader. State: Replica: c05450364e9e4578a1fa7034f7c23b2b, State: Running, Role: LEADER
I20260618 23:31:38.401778 2676 consensus_queue.cc:237] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:38.402438 2124 catalog_manager.cc:5495] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b reported cstate change: term changed from 0 to 1, leader changed from <none> to c05450364e9e4578a1fa7034f7c23b2b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c05450364e9e4578a1fa7034f7c23b2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } health_report { overall_health: UNKNOWN } } }
I20260618 23:31:38.415513 2563 raft_consensus.cc:1261] T abc69b719e874d9ba82e38bc96d0850a P faef863dbed84e0b927b5833deacec3d [term 1 FOLLOWER]: Refusing update from remote peer c05450364e9e4578a1fa7034f7c23b2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:31:38.415711 2423 raft_consensus.cc:1261] T abc69b719e874d9ba82e38bc96d0850a P 21e9603236b94e92b1695c9a1ed5c89d [term 1 FOLLOWER]: Refusing update from remote peer c05450364e9e4578a1fa7034f7c23b2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:31:38.415848 2642 consensus_queue.cc:1035] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:31:38.416231 2642 consensus_queue.cc:1035] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260618 23:31:38.423924 2661 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260618 23:31:39.949525 2124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:31:39.949647 2124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57298:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:31:39.950778 2124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918: Sending DeleteTablet for 3 replicas of tablet abc69b719e874d9ba82e38bc96d0850a
I20260618 23:31:39.951087 2263 tablet_service.cc:1514] Processing DeleteTablet for tablet abc69b719e874d9ba82e38bc96d0850a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36794
I20260618 23:31:39.951220 2403 tablet_service.cc:1514] Processing DeleteTablet for tablet abc69b719e874d9ba82e38bc96d0850a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38124
I20260618 23:31:39.951432 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet abc69b719e874d9ba82e38bc96d0850a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53020
I20260618 23:31:39.951704 2712 tablet_replica.cc:330] T abc69b719e874d9ba82e38bc96d0850a P faef863dbed84e0b927b5833deacec3d: stopping tablet replica
I20260618 23:31:39.951803 2712 raft_consensus.cc:2229] T abc69b719e874d9ba82e38bc96d0850a P faef863dbed84e0b927b5833deacec3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:31:39.951889 2712 raft_consensus.cc:2258] T abc69b719e874d9ba82e38bc96d0850a P faef863dbed84e0b927b5833deacec3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:39.952229 2712 ts_tablet_manager.cc:1905] T abc69b719e874d9ba82e38bc96d0850a P faef863dbed84e0b927b5833deacec3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:31:39.953804 2711 tablet_replica.cc:330] T abc69b719e874d9ba82e38bc96d0850a P 21e9603236b94e92b1695c9a1ed5c89d: stopping tablet replica
I20260618 23:31:39.953908 2711 raft_consensus.cc:2229] T abc69b719e874d9ba82e38bc96d0850a P 21e9603236b94e92b1695c9a1ed5c89d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:31:39.953969 2711 raft_consensus.cc:2258] T abc69b719e874d9ba82e38bc96d0850a P 21e9603236b94e92b1695c9a1ed5c89d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:39.954118 2710 tablet_replica.cc:330] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b: stopping tablet replica
I20260618 23:31:39.954121 2712 ts_tablet_manager.cc:1918] T abc69b719e874d9ba82e38bc96d0850a P faef863dbed84e0b927b5833deacec3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 23:31:39.954182 2712 log.cc:1196] T abc69b719e874d9ba82e38bc96d0850a P faef863dbed84e0b927b5833deacec3d: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/abc69b719e874d9ba82e38bc96d0850a
I20260618 23:31:39.954187 2710 raft_consensus.cc:2229] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:31:39.954288 2711 ts_tablet_manager.cc:1905] T abc69b719e874d9ba82e38bc96d0850a P 21e9603236b94e92b1695c9a1ed5c89d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:31:39.954416 2712 ts_tablet_manager.cc:1939] T abc69b719e874d9ba82e38bc96d0850a P faef863dbed84e0b927b5833deacec3d: Deleting consensus metadata
I20260618 23:31:39.954864 2113 catalog_manager.cc:4841] TS faef863dbed84e0b927b5833deacec3d (127.0.98.3:45889): tablet abc69b719e874d9ba82e38bc96d0850a (table test-table [id=a6bd2310584f49d5a1fcf7b952369958]) successfully deleted
I20260618 23:31:39.955992 2711 ts_tablet_manager.cc:1918] T abc69b719e874d9ba82e38bc96d0850a P 21e9603236b94e92b1695c9a1ed5c89d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 23:31:39.956067 2711 log.cc:1196] T abc69b719e874d9ba82e38bc96d0850a P 21e9603236b94e92b1695c9a1ed5c89d: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/abc69b719e874d9ba82e38bc96d0850a
I20260618 23:31:39.956282 2711 ts_tablet_manager.cc:1939] T abc69b719e874d9ba82e38bc96d0850a P 21e9603236b94e92b1695c9a1ed5c89d: Deleting consensus metadata
I20260618 23:31:39.956643 2113 catalog_manager.cc:4841] TS 21e9603236b94e92b1695c9a1ed5c89d (127.0.98.2:39127): tablet abc69b719e874d9ba82e38bc96d0850a (table test-table [id=a6bd2310584f49d5a1fcf7b952369958]) successfully deleted
I20260618 23:31:39.957973 2710 raft_consensus.cc:2258] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:39.958313 2710 ts_tablet_manager.cc:1905] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:31:39.959930 2710 ts_tablet_manager.cc:1918] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 23:31:39.959987 2710 log.cc:1196] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-0/wal/wals/abc69b719e874d9ba82e38bc96d0850a
I20260618 23:31:39.960230 2710 ts_tablet_manager.cc:1939] T abc69b719e874d9ba82e38bc96d0850a P c05450364e9e4578a1fa7034f7c23b2b: Deleting consensus metadata
I20260618 23:31:39.960616 2112 catalog_manager.cc:4841] TS c05450364e9e4578a1fa7034f7c23b2b (127.0.98.1:42811): tablet abc69b719e874d9ba82e38bc96d0850a (table test-table [id=a6bd2310584f49d5a1fcf7b952369958]) successfully deleted
Jun 18 23:31:39 dist-test-slave-6224 krb5kdc[2081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825499, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 23:31:39 dist-test-slave-6224 krb5kdc[2081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825499, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:39.979756 2720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:39.971163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57326 (local address 127.0.98.62:44909)
0618 23:31:39.971345 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:39.971347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:39.971382 (+ 35us) server_negotiation.cc:407] Connection header received
0618 23:31:39.971499 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:39.971499 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:39.971512 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:39.971704 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:39.972287 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:39.972864 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:39.973643 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:39.973860 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:39.973922 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:39.976526 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:39.976528 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:39.976528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:39.976537 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:39.978258 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:39.978690 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:39.978691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:39.978692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:39.978759 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:39.978993 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:39.978993 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:39.978994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:39.979117 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:39.979205 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:39.979446 (+ 241us) server_negotiation.cc:299] Negotiation successful
0618 23:31:39.979537 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":111,"threads_started":1}
I20260618 23:31:39.981917 2124 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:31:39.982175 2124 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:31:39.984282 2403 tablet_service.cc:1467] Processing CreateTablet for tablet 0de12dd44bcc46aba48e97698f2d4660 (DEFAULT_TABLE table=test-table [id=3a3142e4a12b425d8841401a273a0a1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:39.984485 2403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0de12dd44bcc46aba48e97698f2d4660. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:39.986209 2721 tablet_bootstrap.cc:492] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d: Bootstrap starting.
I20260618 23:31:39.986239 2263 tablet_service.cc:1467] Processing CreateTablet for tablet 0de12dd44bcc46aba48e97698f2d4660 (DEFAULT_TABLE table=test-table [id=3a3142e4a12b425d8841401a273a0a1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:39.986384 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0de12dd44bcc46aba48e97698f2d4660. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:39.986728 2721 tablet_bootstrap.cc:654] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:39.987495 2721 tablet_bootstrap.cc:492] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d: No bootstrap required, opened a new log
I20260618 23:31:39.987641 2721 ts_tablet_manager.cc:1397] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:39.987793 2721 raft_consensus.cc:348] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:39.987931 2721 raft_consensus.cc:374] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:39.987958 2721 raft_consensus.cc:729] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21e9603236b94e92b1695c9a1ed5c89d, State: Initialized, Role: FOLLOWER
I20260618 23:31:39.988018 2721 consensus_queue.cc:260] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:39.988168 2721 ts_tablet_manager.cc:1428] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:39.988173 2543 tablet_service.cc:1467] Processing CreateTablet for tablet 0de12dd44bcc46aba48e97698f2d4660 (DEFAULT_TABLE table=test-table [id=3a3142e4a12b425d8841401a273a0a1c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:39.988314 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0de12dd44bcc46aba48e97698f2d4660. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:39.989804 2724 tablet_bootstrap.cc:492] T 0de12dd44bcc46aba48e97698f2d4660 P faef863dbed84e0b927b5833deacec3d: Bootstrap starting.
I20260618 23:31:39.989820 2723 tablet_bootstrap.cc:492] T 0de12dd44bcc46aba48e97698f2d4660 P c05450364e9e4578a1fa7034f7c23b2b: Bootstrap starting.
I20260618 23:31:39.990289 2723 tablet_bootstrap.cc:654] T 0de12dd44bcc46aba48e97698f2d4660 P c05450364e9e4578a1fa7034f7c23b2b: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:39.990314 2724 tablet_bootstrap.cc:654] T 0de12dd44bcc46aba48e97698f2d4660 P faef863dbed84e0b927b5833deacec3d: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:39.991027 2723 tablet_bootstrap.cc:492] T 0de12dd44bcc46aba48e97698f2d4660 P c05450364e9e4578a1fa7034f7c23b2b: No bootstrap required, opened a new log
I20260618 23:31:39.991035 2724 tablet_bootstrap.cc:492] T 0de12dd44bcc46aba48e97698f2d4660 P faef863dbed84e0b927b5833deacec3d: No bootstrap required, opened a new log
I20260618 23:31:39.991080 2723 ts_tablet_manager.cc:1397] T 0de12dd44bcc46aba48e97698f2d4660 P c05450364e9e4578a1fa7034f7c23b2b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:39.991102 2724 ts_tablet_manager.cc:1397] T 0de12dd44bcc46aba48e97698f2d4660 P faef863dbed84e0b927b5833deacec3d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:39.991228 2723 raft_consensus.cc:348] T 0de12dd44bcc46aba48e97698f2d4660 P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:39.991314 2723 raft_consensus.cc:374] T 0de12dd44bcc46aba48e97698f2d4660 P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:39.991258 2724 raft_consensus.cc:348] T 0de12dd44bcc46aba48e97698f2d4660 P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:39.991330 2723 raft_consensus.cc:729] T 0de12dd44bcc46aba48e97698f2d4660 P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05450364e9e4578a1fa7034f7c23b2b, State: Initialized, Role: FOLLOWER
I20260618 23:31:39.991346 2724 raft_consensus.cc:374] T 0de12dd44bcc46aba48e97698f2d4660 P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:39.991402 2724 raft_consensus.cc:729] T 0de12dd44bcc46aba48e97698f2d4660 P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faef863dbed84e0b927b5833deacec3d, State: Initialized, Role: FOLLOWER
I20260618 23:31:39.991397 2723 consensus_queue.cc:260] T 0de12dd44bcc46aba48e97698f2d4660 P c05450364e9e4578a1fa7034f7c23b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:39.991498 2723 ts_tablet_manager.cc:1428] T 0de12dd44bcc46aba48e97698f2d4660 P c05450364e9e4578a1fa7034f7c23b2b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260618 23:31:39.991487 2724 consensus_queue.cc:260] T 0de12dd44bcc46aba48e97698f2d4660 P faef863dbed84e0b927b5833deacec3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:39.991719 2724 ts_tablet_manager.cc:1428] T 0de12dd44bcc46aba48e97698f2d4660 P faef863dbed84e0b927b5833deacec3d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:40.411713 2692 raft_consensus.cc:482] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:31:40.411868 2692 raft_consensus.cc:504] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:40.412097 2692 leader_election.cc:288] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c05450364e9e4578a1fa7034f7c23b2b (127.0.98.1:42811), faef863dbed84e0b927b5833deacec3d (127.0.98.3:45889)
I20260618 23:31:40.412397 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0de12dd44bcc46aba48e97698f2d4660" candidate_uuid: "21e9603236b94e92b1695c9a1ed5c89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faef863dbed84e0b927b5833deacec3d" is_pre_election: true
I20260618 23:31:40.412424 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0de12dd44bcc46aba48e97698f2d4660" candidate_uuid: "21e9603236b94e92b1695c9a1ed5c89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05450364e9e4578a1fa7034f7c23b2b" is_pre_election: true
I20260618 23:31:40.412549 2563 raft_consensus.cc:2454] T 0de12dd44bcc46aba48e97698f2d4660 P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21e9603236b94e92b1695c9a1ed5c89d in term 0.
I20260618 23:31:40.412549 2283 raft_consensus.cc:2454] T 0de12dd44bcc46aba48e97698f2d4660 P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21e9603236b94e92b1695c9a1ed5c89d in term 0.
I20260618 23:31:40.412774 2356 leader_election.cc:302] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21e9603236b94e92b1695c9a1ed5c89d, c05450364e9e4578a1fa7034f7c23b2b; no voters:
I20260618 23:31:40.412961 2692 raft_consensus.cc:2781] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:31:40.413012 2692 raft_consensus.cc:482] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:31:40.413033 2692 raft_consensus.cc:3037] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:40.413900 2692 raft_consensus.cc:504] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:40.414166 2692 leader_election.cc:288] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d [CANDIDATE]: Term 1 election: Requested vote from peers c05450364e9e4578a1fa7034f7c23b2b (127.0.98.1:42811), faef863dbed84e0b927b5833deacec3d (127.0.98.3:45889)
I20260618 23:31:40.414214 2283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0de12dd44bcc46aba48e97698f2d4660" candidate_uuid: "21e9603236b94e92b1695c9a1ed5c89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c05450364e9e4578a1fa7034f7c23b2b"
I20260618 23:31:40.414302 2283 raft_consensus.cc:3037] T 0de12dd44bcc46aba48e97698f2d4660 P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:40.414386 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0de12dd44bcc46aba48e97698f2d4660" candidate_uuid: "21e9603236b94e92b1695c9a1ed5c89d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faef863dbed84e0b927b5833deacec3d"
I20260618 23:31:40.414510 2563 raft_consensus.cc:3037] T 0de12dd44bcc46aba48e97698f2d4660 P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:40.415037 2283 raft_consensus.cc:2454] T 0de12dd44bcc46aba48e97698f2d4660 P c05450364e9e4578a1fa7034f7c23b2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21e9603236b94e92b1695c9a1ed5c89d in term 1.
I20260618 23:31:40.415222 2356 leader_election.cc:302] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21e9603236b94e92b1695c9a1ed5c89d, c05450364e9e4578a1fa7034f7c23b2b; no voters:
I20260618 23:31:40.415328 2692 raft_consensus.cc:2781] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:40.415392 2692 raft_consensus.cc:686] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d [term 1 LEADER]: Becoming Leader. State: Replica: 21e9603236b94e92b1695c9a1ed5c89d, State: Running, Role: LEADER
I20260618 23:31:40.415450 2692 consensus_queue.cc:237] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:40.415474 2563 raft_consensus.cc:2454] T 0de12dd44bcc46aba48e97698f2d4660 P faef863dbed84e0b927b5833deacec3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21e9603236b94e92b1695c9a1ed5c89d in term 1.
I20260618 23:31:40.416289 2126 catalog_manager.cc:5495] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d reported cstate change: term changed from 0 to 1, leader changed from <none> to 21e9603236b94e92b1695c9a1ed5c89d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "21e9603236b94e92b1695c9a1ed5c89d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } health_report { overall_health: UNKNOWN } } }
I20260618 23:31:40.435771 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:40.433193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38144 (local address 127.0.98.2:39127)
0618 23:31:40.433426 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:40.433428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:40.433438 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:31:40.433468 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:40.433469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:40.433479 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:40.433548 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:31:40.433687 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:40.434278 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:40.435097 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:40.435297 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:40.435369 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:40.435480 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:31:40.435509 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:31:40.435567 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:40.435684 (+ 117us) server_negotiation.cc:299] Negotiation successful
0618 23:31:40.435703 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":165,"threads_started":1}
I20260618 23:31:40.440740 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:40.438360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57097 (local address 127.0.98.2:39127)
0618 23:31:40.438446 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:40.438448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:40.438460 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:31:40.438541 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:40.438542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:40.438554 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:40.438613 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:31:40.438722 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:40.439217 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:40.440056 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:40.440250 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:40.440307 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:40.440443 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:31:40.440470 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:31:40.440510 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:40.440590 (+ 80us) server_negotiation.cc:299] Negotiation successful
0618 23:31:40.440610 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260618 23:31:40.440742 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:40.438276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39127 (local address 127.0.98.2:57097)
0618 23:31:40.438400 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:40.438411 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:40.438464 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:40.438618 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:40.438618 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:40.438621 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:31:40.438661 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:40.438662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:40.439302 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:40.439303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:40.439974 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:40.439976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:40.440327 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:40.440328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:40.440368 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:40.440373 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:31:40.440529 (+ 156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:31:40.440530 (+ 1us) client_negotiation.cc:769] Sending connection context
0618 23:31:40.440570 (+ 40us) client_negotiation.cc:240] Negotiation successful
0618 23:31:40.440608 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":79,"threads_started":1}
I20260618 23:31:40.443317 2563 raft_consensus.cc:1261] T 0de12dd44bcc46aba48e97698f2d4660 P faef863dbed84e0b927b5833deacec3d [term 1 FOLLOWER]: Refusing update from remote peer 21e9603236b94e92b1695c9a1ed5c89d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:31:40.443410 2283 raft_consensus.cc:1261] T 0de12dd44bcc46aba48e97698f2d4660 P c05450364e9e4578a1fa7034f7c23b2b [term 1 FOLLOWER]: Refusing update from remote peer 21e9603236b94e92b1695c9a1ed5c89d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:31:40.443594 2654 consensus_queue.cc:1035] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d [LEADER]: Connected to new peer: Peer: permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:31:40.443724 2697 consensus_queue.cc:1035] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:31:40.460337 2126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:31:40.460490 2126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:31:40.461433 2126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918: Sending DeleteTablet for 3 replicas of tablet 0de12dd44bcc46aba48e97698f2d4660
I20260618 23:31:40.461864 2263 tablet_service.cc:1514] Processing DeleteTablet for tablet 0de12dd44bcc46aba48e97698f2d4660 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36794
I20260618 23:31:40.461935 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet 0de12dd44bcc46aba48e97698f2d4660 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53020
I20260618 23:31:40.462114 2738 tablet_replica.cc:330] T 0de12dd44bcc46aba48e97698f2d4660 P c05450364e9e4578a1fa7034f7c23b2b: stopping tablet replica
I20260618 23:31:40.462203 2738 raft_consensus.cc:2229] T 0de12dd44bcc46aba48e97698f2d4660 P c05450364e9e4578a1fa7034f7c23b2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:31:40.462258 2738 raft_consensus.cc:2258] T 0de12dd44bcc46aba48e97698f2d4660 P c05450364e9e4578a1fa7034f7c23b2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:40.462661 2738 ts_tablet_manager.cc:1905] T 0de12dd44bcc46aba48e97698f2d4660 P c05450364e9e4578a1fa7034f7c23b2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:31:40.462571 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:31:40.462769 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:31:40.463090 2739 tablet_replica.cc:330] T 0de12dd44bcc46aba48e97698f2d4660 P faef863dbed84e0b927b5833deacec3d: stopping tablet replica
I20260618 23:31:40.463071 2402 tablet_service.cc:1514] Processing DeleteTablet for tablet 0de12dd44bcc46aba48e97698f2d4660 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38124
I20260618 23:31:40.463176 2739 raft_consensus.cc:2229] T 0de12dd44bcc46aba48e97698f2d4660 P faef863dbed84e0b927b5833deacec3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:31:40.463238 2739 raft_consensus.cc:2258] T 0de12dd44bcc46aba48e97698f2d4660 P faef863dbed84e0b927b5833deacec3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:40.463696 2739 ts_tablet_manager.cc:1905] T 0de12dd44bcc46aba48e97698f2d4660 P faef863dbed84e0b927b5833deacec3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:31:40.464872 2263 tablet_service.cc:1467] Processing CreateTablet for tablet 41ec98c696e74fb1809f5817950e0d7b (DEFAULT_TABLE table=test-table [id=ed8c9da5ed4f4d628ef52c3da9cf598c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:40.464978 2402 tablet_service.cc:1467] Processing CreateTablet for tablet 41ec98c696e74fb1809f5817950e0d7b (DEFAULT_TABLE table=test-table [id=ed8c9da5ed4f4d628ef52c3da9cf598c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:40.465114 2402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ec98c696e74fb1809f5817950e0d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:40.465595 2739 ts_tablet_manager.cc:1918] T 0de12dd44bcc46aba48e97698f2d4660 P faef863dbed84e0b927b5833deacec3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:31:40.465668 2739 log.cc:1196] T 0de12dd44bcc46aba48e97698f2d4660 P faef863dbed84e0b927b5833deacec3d: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/0de12dd44bcc46aba48e97698f2d4660
I20260618 23:31:40.465939 2739 ts_tablet_manager.cc:1939] T 0de12dd44bcc46aba48e97698f2d4660 P faef863dbed84e0b927b5833deacec3d: Deleting consensus metadata
I20260618 23:31:40.466351 2113 catalog_manager.cc:4841] TS faef863dbed84e0b927b5833deacec3d (127.0.98.3:45889): tablet 0de12dd44bcc46aba48e97698f2d4660 (table test-table [id=3a3142e4a12b425d8841401a273a0a1c]) successfully deleted
I20260618 23:31:40.466794 2721 tablet_bootstrap.cc:492] T 41ec98c696e74fb1809f5817950e0d7b P 21e9603236b94e92b1695c9a1ed5c89d: Bootstrap starting.
I20260618 23:31:40.466822 2543 tablet_service.cc:1467] Processing CreateTablet for tablet 41ec98c696e74fb1809f5817950e0d7b (DEFAULT_TABLE table=test-table [id=ed8c9da5ed4f4d628ef52c3da9cf598c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:40.466958 2543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ec98c696e74fb1809f5817950e0d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:40.467233 2721 tablet_bootstrap.cc:654] T 41ec98c696e74fb1809f5817950e0d7b P 21e9603236b94e92b1695c9a1ed5c89d: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:40.468096 2721 tablet_bootstrap.cc:492] T 41ec98c696e74fb1809f5817950e0d7b P 21e9603236b94e92b1695c9a1ed5c89d: No bootstrap required, opened a new log
I20260618 23:31:40.468156 2721 ts_tablet_manager.cc:1397] T 41ec98c696e74fb1809f5817950e0d7b P 21e9603236b94e92b1695c9a1ed5c89d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:40.468334 2721 raft_consensus.cc:348] T 41ec98c696e74fb1809f5817950e0d7b P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:40.468416 2721 raft_consensus.cc:374] T 41ec98c696e74fb1809f5817950e0d7b P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:40.468441 2721 raft_consensus.cc:729] T 41ec98c696e74fb1809f5817950e0d7b P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21e9603236b94e92b1695c9a1ed5c89d, State: Initialized, Role: FOLLOWER
I20260618 23:31:40.468480 2724 tablet_bootstrap.cc:492] T 41ec98c696e74fb1809f5817950e0d7b P faef863dbed84e0b927b5833deacec3d: Bootstrap starting.
I20260618 23:31:40.468495 2721 consensus_queue.cc:260] T 41ec98c696e74fb1809f5817950e0d7b P 21e9603236b94e92b1695c9a1ed5c89d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:40.468611 2721 ts_tablet_manager.cc:1428] T 41ec98c696e74fb1809f5817950e0d7b P 21e9603236b94e92b1695c9a1ed5c89d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:40.468694 2263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ec98c696e74fb1809f5817950e0d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:40.468899 2724 tablet_bootstrap.cc:654] T 41ec98c696e74fb1809f5817950e0d7b P faef863dbed84e0b927b5833deacec3d: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:40.469609 2724 tablet_bootstrap.cc:492] T 41ec98c696e74fb1809f5817950e0d7b P faef863dbed84e0b927b5833deacec3d: No bootstrap required, opened a new log
I20260618 23:31:40.469673 2724 ts_tablet_manager.cc:1397] T 41ec98c696e74fb1809f5817950e0d7b P faef863dbed84e0b927b5833deacec3d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:40.469877 2724 raft_consensus.cc:348] T 41ec98c696e74fb1809f5817950e0d7b P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:40.469971 2724 raft_consensus.cc:374] T 41ec98c696e74fb1809f5817950e0d7b P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:40.469995 2724 raft_consensus.cc:729] T 41ec98c696e74fb1809f5817950e0d7b P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faef863dbed84e0b927b5833deacec3d, State: Initialized, Role: FOLLOWER
I20260618 23:31:40.470048 2724 consensus_queue.cc:260] T 41ec98c696e74fb1809f5817950e0d7b P faef863dbed84e0b927b5833deacec3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:40.470139 2724 ts_tablet_manager.cc:1428] T 41ec98c696e74fb1809f5817950e0d7b P faef863dbed84e0b927b5833deacec3d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:40.470435 2723 tablet_bootstrap.cc:492] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b: Bootstrap starting.
I20260618 23:31:40.470871 2723 tablet_bootstrap.cc:654] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:40.471494 2738 ts_tablet_manager.cc:1918] T 0de12dd44bcc46aba48e97698f2d4660 P c05450364e9e4578a1fa7034f7c23b2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:31:40.471616 2723 tablet_bootstrap.cc:492] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b: No bootstrap required, opened a new log
I20260618 23:31:40.471692 2740 tablet_replica.cc:330] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d: stopping tablet replica
I20260618 23:31:40.471698 2723 ts_tablet_manager.cc:1397] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:40.471771 2740 raft_consensus.cc:2229] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:31:40.471835 2723 raft_consensus.cc:348] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:40.471927 2723 raft_consensus.cc:374] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:40.471915 2740 raft_consensus.cc:2258] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:40.471953 2723 raft_consensus.cc:729] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c05450364e9e4578a1fa7034f7c23b2b, State: Initialized, Role: FOLLOWER
I20260618 23:31:40.472009 2723 consensus_queue.cc:260] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:40.472220 2723 ts_tablet_manager.cc:1428] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:40.471616 2738 log.cc:1196] T 0de12dd44bcc46aba48e97698f2d4660 P c05450364e9e4578a1fa7034f7c23b2b: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-0/wal/wals/0de12dd44bcc46aba48e97698f2d4660
I20260618 23:31:40.472595 2738 ts_tablet_manager.cc:1939] T 0de12dd44bcc46aba48e97698f2d4660 P c05450364e9e4578a1fa7034f7c23b2b: Deleting consensus metadata
I20260618 23:31:40.472960 2112 catalog_manager.cc:4841] TS c05450364e9e4578a1fa7034f7c23b2b (127.0.98.1:42811): tablet 0de12dd44bcc46aba48e97698f2d4660 (table test-table [id=3a3142e4a12b425d8841401a273a0a1c]) successfully deleted
I20260618 23:31:40.473201 2740 ts_tablet_manager.cc:1905] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:31:40.475003 2740 ts_tablet_manager.cc:1918] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:31:40.475082 2740 log.cc:1196] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/0de12dd44bcc46aba48e97698f2d4660
I20260618 23:31:40.475327 2740 ts_tablet_manager.cc:1939] T 0de12dd44bcc46aba48e97698f2d4660 P 21e9603236b94e92b1695c9a1ed5c89d: Deleting consensus metadata
I20260618 23:31:40.475756 2113 catalog_manager.cc:4841] TS 21e9603236b94e92b1695c9a1ed5c89d (127.0.98.2:39127): tablet 0de12dd44bcc46aba48e97698f2d4660 (table test-table [id=3a3142e4a12b425d8841401a273a0a1c]) successfully deleted
I20260618 23:31:40.532330 2689 raft_consensus.cc:482] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:31:40.532511 2689 raft_consensus.cc:504] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:40.532804 2689 leader_election.cc:288] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21e9603236b94e92b1695c9a1ed5c89d (127.0.98.2:39127), faef863dbed84e0b927b5833deacec3d (127.0.98.3:45889)
I20260618 23:31:40.533082 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ec98c696e74fb1809f5817950e0d7b" candidate_uuid: "c05450364e9e4578a1fa7034f7c23b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21e9603236b94e92b1695c9a1ed5c89d" is_pre_election: true
I20260618 23:31:40.533233 2423 raft_consensus.cc:2454] T 41ec98c696e74fb1809f5817950e0d7b P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05450364e9e4578a1fa7034f7c23b2b in term 0.
I20260618 23:31:40.533159 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ec98c696e74fb1809f5817950e0d7b" candidate_uuid: "c05450364e9e4578a1fa7034f7c23b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faef863dbed84e0b927b5833deacec3d" is_pre_election: true
I20260618 23:31:40.533318 2563 raft_consensus.cc:2454] T 41ec98c696e74fb1809f5817950e0d7b P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c05450364e9e4578a1fa7034f7c23b2b in term 0.
I20260618 23:31:40.533493 2217 leader_election.cc:302] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21e9603236b94e92b1695c9a1ed5c89d, c05450364e9e4578a1fa7034f7c23b2b; no voters:
I20260618 23:31:40.533672 2689 raft_consensus.cc:2781] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:31:40.533738 2689 raft_consensus.cc:482] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:31:40.533763 2689 raft_consensus.cc:3037] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:40.534757 2689 raft_consensus.cc:504] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:40.534921 2689 leader_election.cc:288] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b [CANDIDATE]: Term 1 election: Requested vote from peers 21e9603236b94e92b1695c9a1ed5c89d (127.0.98.2:39127), faef863dbed84e0b927b5833deacec3d (127.0.98.3:45889)
I20260618 23:31:40.535131 2423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ec98c696e74fb1809f5817950e0d7b" candidate_uuid: "c05450364e9e4578a1fa7034f7c23b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21e9603236b94e92b1695c9a1ed5c89d"
I20260618 23:31:40.535218 2423 raft_consensus.cc:3037] T 41ec98c696e74fb1809f5817950e0d7b P 21e9603236b94e92b1695c9a1ed5c89d [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:40.535183 2563 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ec98c696e74fb1809f5817950e0d7b" candidate_uuid: "c05450364e9e4578a1fa7034f7c23b2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faef863dbed84e0b927b5833deacec3d"
I20260618 23:31:40.535292 2563 raft_consensus.cc:3037] T 41ec98c696e74fb1809f5817950e0d7b P faef863dbed84e0b927b5833deacec3d [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:40.536141 2423 raft_consensus.cc:2454] T 41ec98c696e74fb1809f5817950e0d7b P 21e9603236b94e92b1695c9a1ed5c89d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05450364e9e4578a1fa7034f7c23b2b in term 1.
I20260618 23:31:40.536319 2563 raft_consensus.cc:2454] T 41ec98c696e74fb1809f5817950e0d7b P faef863dbed84e0b927b5833deacec3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c05450364e9e4578a1fa7034f7c23b2b in term 1.
I20260618 23:31:40.536384 2217 leader_election.cc:302] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21e9603236b94e92b1695c9a1ed5c89d, c05450364e9e4578a1fa7034f7c23b2b; no voters:
I20260618 23:31:40.536494 2689 raft_consensus.cc:2781] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:40.536634 2689 raft_consensus.cc:686] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b [term 1 LEADER]: Becoming Leader. State: Replica: c05450364e9e4578a1fa7034f7c23b2b, State: Running, Role: LEADER
I20260618 23:31:40.536710 2689 consensus_queue.cc:237] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } }
I20260618 23:31:40.537488 2126 catalog_manager.cc:5495] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b reported cstate change: term changed from 0 to 1, leader changed from <none> to c05450364e9e4578a1fa7034f7c23b2b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c05450364e9e4578a1fa7034f7c23b2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c05450364e9e4578a1fa7034f7c23b2b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42811 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 } health_report { overall_health: UNKNOWN } } }
I20260618 23:31:40.565414 2746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:40.562010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36814 (local address 127.0.98.1:42811)
0618 23:31:40.562206 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:40.562208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:40.562213 (+ 5us) server_negotiation.cc:407] Connection header received
0618 23:31:40.562231 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:40.562232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:40.562243 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:40.562303 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:31:40.562548 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:40.563154 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:40.564334 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:40.564739 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:40.564840 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:40.565107 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:31:40.565140 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:31:40.565188 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:40.565293 (+ 105us) server_negotiation.cc:299] Negotiation successful
0618 23:31:40.565313 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":129,"threads_started":1}
I20260618 23:31:40.568045 2563 raft_consensus.cc:1261] T 41ec98c696e74fb1809f5817950e0d7b P faef863dbed84e0b927b5833deacec3d [term 1 FOLLOWER]: Refusing update from remote peer c05450364e9e4578a1fa7034f7c23b2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:31:40.568044 2423 raft_consensus.cc:1261] T 41ec98c696e74fb1809f5817950e0d7b P 21e9603236b94e92b1695c9a1ed5c89d [term 1 FOLLOWER]: Refusing update from remote peer c05450364e9e4578a1fa7034f7c23b2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:31:40.568400 2744 consensus_queue.cc:1035] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "21e9603236b94e92b1695c9a1ed5c89d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:31:40.568681 2689 consensus_queue.cc:1035] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "faef863dbed84e0b927b5833deacec3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260618 23:31:40.575286 2694 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260618 23:31:42.103682 2124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:31:42.103823 2124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:31:42.105077 2124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6a6c7c3b43649378d624394efbe9918: Sending DeleteTablet for 3 replicas of tablet 41ec98c696e74fb1809f5817950e0d7b
I20260618 23:31:42.105355 2263 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ec98c696e74fb1809f5817950e0d7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36794
I20260618 23:31:42.105406 2402 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ec98c696e74fb1809f5817950e0d7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38124
I20260618 23:31:42.105450 2543 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ec98c696e74fb1809f5817950e0d7b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53020
I20260618 23:31:42.105643 2773 tablet_replica.cc:330] T 41ec98c696e74fb1809f5817950e0d7b P 21e9603236b94e92b1695c9a1ed5c89d: stopping tablet replica
I20260618 23:31:42.105659 2771 tablet_replica.cc:330] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b: stopping tablet replica
I20260618 23:31:42.105710 2773 raft_consensus.cc:2229] T 41ec98c696e74fb1809f5817950e0d7b P 21e9603236b94e92b1695c9a1ed5c89d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:31:42.105711 2771 raft_consensus.cc:2229] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:31:42.105760 2773 raft_consensus.cc:2258] T 41ec98c696e74fb1809f5817950e0d7b P 21e9603236b94e92b1695c9a1ed5c89d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:42.105788 2771 raft_consensus.cc:2258] T 41ec98c696e74fb1809f5817950e0d7b P c05450364e9e4578a1fa7034f7c23b2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:42.105795 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 2198
I20260618 23:31:42.106478 2772 tablet_replica.cc:330] T 41ec98c696e74fb1809f5817950e0d7b P faef863dbed84e0b927b5833deacec3d: stopping tablet replica
I20260618 23:31:42.106540 2772 raft_consensus.cc:2229] T 41ec98c696e74fb1809f5817950e0d7b P faef863dbed84e0b927b5833deacec3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:31:42.106580 2772 raft_consensus.cc:2258] T 41ec98c696e74fb1809f5817950e0d7b P faef863dbed84e0b927b5833deacec3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:42.107052 2773 ts_tablet_manager.cc:1905] T 41ec98c696e74fb1809f5817950e0d7b P 21e9603236b94e92b1695c9a1ed5c89d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:31:42.108273 2772 ts_tablet_manager.cc:1905] T 41ec98c696e74fb1809f5817950e0d7b P faef863dbed84e0b927b5833deacec3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:31:42.109071 2773 ts_tablet_manager.cc:1918] T 41ec98c696e74fb1809f5817950e0d7b P 21e9603236b94e92b1695c9a1ed5c89d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 23:31:42.109193 2773 log.cc:1196] T 41ec98c696e74fb1809f5817950e0d7b P 21e9603236b94e92b1695c9a1ed5c89d: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/41ec98c696e74fb1809f5817950e0d7b
I20260618 23:31:42.109504 2773 ts_tablet_manager.cc:1939] T 41ec98c696e74fb1809f5817950e0d7b P 21e9603236b94e92b1695c9a1ed5c89d: Deleting consensus metadata
I20260618 23:31:42.109974 2113 catalog_manager.cc:4841] TS 21e9603236b94e92b1695c9a1ed5c89d (127.0.98.2:39127): tablet 41ec98c696e74fb1809f5817950e0d7b (table test-table [id=ed8c9da5ed4f4d628ef52c3da9cf598c]) successfully deleted
I20260618 23:31:42.110023 2772 ts_tablet_manager.cc:1918] T 41ec98c696e74fb1809f5817950e0d7b P faef863dbed84e0b927b5833deacec3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 23:31:42.110069 2772 log.cc:1196] T 41ec98c696e74fb1809f5817950e0d7b P faef863dbed84e0b927b5833deacec3d: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/41ec98c696e74fb1809f5817950e0d7b
I20260618 23:31:42.110294 2772 ts_tablet_manager.cc:1939] T 41ec98c696e74fb1809f5817950e0d7b P faef863dbed84e0b927b5833deacec3d: Deleting consensus metadata
I20260618 23:31:42.110834 2113 catalog_manager.cc:4841] TS faef863dbed84e0b927b5833deacec3d (127.0.98.3:45889): tablet 41ec98c696e74fb1809f5817950e0d7b (table test-table [id=ed8c9da5ed4f4d628ef52c3da9cf598c]) successfully deleted
W20260618 23:31:42.114183 2112 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42811 (error 108)
I20260618 23:31:42.114542 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 2338
I20260618 23:31:42.116699 2775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:42.116384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42811 (local address 127.0.0.1:36830)
0618 23:31:42.116558 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:42.116649 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
W20260618 23:31:42.116851 2112 catalog_manager.cc:4584] TS c05450364e9e4578a1fa7034f7c23b2b (127.0.98.1:42811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 41ec98c696e74fb1809f5817950e0d7b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42811: connect: Connection refused (error 111)
I20260618 23:31:42.121718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 2478
I20260618 23:31:42.127329 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 2094
2026-06-18T23:31:42Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5987 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:31:42 dist-test-slave-6224 krb5kdc[2778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:31:42 dist-test-slave-6224 krb5kdc[2778](info): set up 2 sockets
Jun 18 23:31:42 dist-test-slave-6224 krb5kdc[2778](info): commencing operation
krb5kdc: starting...
W20260618 23:31:43.184017 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:31:43 dist-test-slave-6224 krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:31:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:31:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:31:43.314895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44289
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44289
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:43.407809 2791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:43.408077 2791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:43.408107 2791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:43.409387 2791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:31:43.409435 2791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:43.409449 2791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:43.409462 2791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:31:43.409475 2791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:31:43.411096 2791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:43.411355 2791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:43.411607 2791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:43.417305 2797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:43.417341 2796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:43.417333 2791 server_base.cc:1034] running on GCE node
W20260618 23:31:43.417301 2799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:43.417989 2791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:43.418438 2791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:43.419633 2791 hybrid_clock.cc:648] HybridClock initialized: now 1781825503419603 us; error 45 us; skew 500 ppm
Jun 18 23:31:43 dist-test-slave-6224 krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:43.422667 2791 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:31:43.423218 2791 webserver.cc:460] Webserver started at http://127.0.98.62:39741/ using document root <none> and password file <none>
I20260618 23:31:43.423491 2791 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:43.423593 2791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:43.423770 2791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:43.424991 2791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "bb5d1b4f648340fba45da6c0c4344cb8"
format_stamp: "Formatted at 2026-06-18 23:31:43 on dist-test-slave-6224"
server_key: "aa67864d5264df3c5d44d83e36b5e0d8"
server_key_iv: "b642a66c7a941b10c30514c43fc3ba62"
server_key_version: "encryptionkey@0"
I20260618 23:31:43.425395 2791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "bb5d1b4f648340fba45da6c0c4344cb8"
format_stamp: "Formatted at 2026-06-18 23:31:43 on dist-test-slave-6224"
server_key: "aa67864d5264df3c5d44d83e36b5e0d8"
server_key_iv: "b642a66c7a941b10c30514c43fc3ba62"
server_key_version: "encryptionkey@0"
I20260618 23:31:43.427006 2791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:31:43.428005 2806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:43.428243 2791 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:43.428366 2791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "bb5d1b4f648340fba45da6c0c4344cb8"
format_stamp: "Formatted at 2026-06-18 23:31:43 on dist-test-slave-6224"
server_key: "aa67864d5264df3c5d44d83e36b5e0d8"
server_key_iv: "b642a66c7a941b10c30514c43fc3ba62"
server_key_version: "encryptionkey@0"
I20260618 23:31:43.428450 2791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:43.457650 2791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:43.460037 2791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:43.460211 2791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:43.465090 2791 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44289
I20260618 23:31:43.465168 2858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44289 every 8 connection(s)
I20260618 23:31:43.465751 2791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:31:43.466598 2859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:43.469352 2859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8: Bootstrap starting.
I20260618 23:31:43.469995 2859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:43.470276 2859 log.cc:824] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:43.471105 2859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8: No bootstrap required, opened a new log
I20260618 23:31:43.471305 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 2791
I20260618 23:31:43.471437 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:31:43.471678 392 external_mini_cluster.cc:1428] Setting key 804dac67784ef516776ef2141c9fcaf2
I20260618 23:31:43.473887 2859 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5d1b4f648340fba45da6c0c4344cb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } }
I20260618 23:31:43.474083 2859 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:43.474120 2859 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb5d1b4f648340fba45da6c0c4344cb8, State: Initialized, Role: FOLLOWER
I20260618 23:31:43.474334 2859 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5d1b4f648340fba45da6c0c4344cb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } }
I20260618 23:31:43.474433 2859 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:31:43.474480 2859 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:31:43.474534 2859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:43.475152 2859 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5d1b4f648340fba45da6c0c4344cb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } }
I20260618 23:31:43.475284 2859 leader_election.cc:302] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb5d1b4f648340fba45da6c0c4344cb8; no voters:
I20260618 23:31:43.475498 2859 leader_election.cc:288] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:31:43.475603 2864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:43.475849 2859 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:31:43.475867 2864 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8 [term 1 LEADER]: Becoming Leader. State: Replica: bb5d1b4f648340fba45da6c0c4344cb8, State: Running, Role: LEADER
I20260618 23:31:43.476274 2864 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5d1b4f648340fba45da6c0c4344cb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } }
I20260618 23:31:43.477816 2865 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb5d1b4f648340fba45da6c0c4344cb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5d1b4f648340fba45da6c0c4344cb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } } }
I20260618 23:31:43.478119 2865 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8 [sys.catalog]: This master's current role is: LEADER
W20260618 23:31:43.478271 2879 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:31:43.478331 2879 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:31:43.477985 2866 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb5d1b4f648340fba45da6c0c4344cb8. Latest consensus state: current_term: 1 leader_uuid: "bb5d1b4f648340fba45da6c0c4344cb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb5d1b4f648340fba45da6c0c4344cb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44289 } } }
I20260618 23:31:43.478418 2866 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8 [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:43.478893 2881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:31:43.479142 2881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 18 23:31:43 dist-test-slave-6224 krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:43.481655 2881 catalog_manager.cc:1347] Generated new cluster ID: ad5d71757dd0487abfd1700c9c3982d6
I20260618 23:31:43.481734 2881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:31:43.484975 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:43.472297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48584 (local address 127.0.98.62:44289)
0618 23:31:43.472508 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:43.472518 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:43.472549 (+ 31us) server_negotiation.cc:407] Connection header received
0618 23:31:43.473914 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:43.473923 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:43.473986 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:43.474175 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:43.474966 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:43.476369 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:43.477701 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:43.478257 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:43.478371 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:43.481392 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:43.481401 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:43.481402 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:43.481414 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:43.483259 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:43.483686 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:43.483689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:43.483689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:43.483816 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:43.484022 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:43.484023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:43.484024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:43.484290 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:43.484498 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:43.484567 (+ 69us) server_negotiation.cc:299] Negotiation successful
0618 23:31:43.484733 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
I20260618 23:31:43.515077 2881 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:31:43.515946 2881 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:31:43.520399 2881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb5d1b4f648340fba45da6c0c4344cb8: Generated new TSK 0
I20260618 23:31:43.520612 2881 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260618 23:31:43.552583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44289
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:43.648393 2885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:43.648681 2885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:43.648725 2885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:43.650135 2885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:43.650207 2885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:43.650254 2885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:31:43.652034 2885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:43.652345 2885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:43.652621 2885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:43.658459 2890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:43.658481 2893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:43.658459 2891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:43.658836 2885 server_base.cc:1034] running on GCE node
I20260618 23:31:43.659088 2885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:43.659361 2885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:43.660527 2885 hybrid_clock.cc:648] HybridClock initialized: now 1781825503660512 us; error 33 us; skew 500 ppm
Jun 18 23:31:43 dist-test-slave-6224 krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:43.663648 2885 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:31:43.664191 2885 webserver.cc:460] Webserver started at http://127.0.98.1:44707/ using document root <none> and password file <none>
I20260618 23:31:43.664464 2885 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:43.664520 2885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:43.664722 2885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:43.665995 2885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "2156571d159643729eb6a8f41a4d7a3d"
format_stamp: "Formatted at 2026-06-18 23:31:43 on dist-test-slave-6224"
server_key: "015149ad5bf7d08efeff557eff023bfd"
server_key_iv: "b9ce3772c79f1044449a8c2b2a346506"
server_key_version: "encryptionkey@0"
I20260618 23:31:43.666425 2885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2156571d159643729eb6a8f41a4d7a3d"
format_stamp: "Formatted at 2026-06-18 23:31:43 on dist-test-slave-6224"
server_key: "015149ad5bf7d08efeff557eff023bfd"
server_key_iv: "b9ce3772c79f1044449a8c2b2a346506"
server_key_version: "encryptionkey@0"
I20260618 23:31:43.668156 2885 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260618 23:31:43.669158 2900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:43.669466 2885 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:43.669585 2885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "2156571d159643729eb6a8f41a4d7a3d"
format_stamp: "Formatted at 2026-06-18 23:31:43 on dist-test-slave-6224"
server_key: "015149ad5bf7d08efeff557eff023bfd"
server_key_iv: "b9ce3772c79f1044449a8c2b2a346506"
server_key_version: "encryptionkey@0"
I20260618 23:31:43.669675 2885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:43.706171 2885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:43.708705 2885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:43.708879 2885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:43.709178 2885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:43.709551 2885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:43.709618 2885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:43.709664 2885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:43.709692 2885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:43.716894 2885 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37547
I20260618 23:31:43.716943 3013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37547 every 8 connection(s)
I20260618 23:31:43.717283 2885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:31:43.719588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 2885
I20260618 23:31:43.719717 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:31:43.719923 392 external_mini_cluster.cc:1428] Setting key 2b7b638771ddfaa4d4d57f54d52811d7
Jun 18 23:31:43 dist-test-slave-6224 krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260618 23:31:43.729852 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:43.718071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58077 (local address 127.0.98.62:44289)
0618 23:31:43.718147 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:43.718149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:43.718370 (+ 221us) server_negotiation.cc:407] Connection header received
0618 23:31:43.719401 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:43.719403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:43.719427 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:43.719455 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:43.720460 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:43.721097 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:43.722181 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:43.722531 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:43.722574 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:43.726255 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:43.726258 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:43.726258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:43.726280 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:43.728172 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:43.728702 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:43.728703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:43.728704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:43.728770 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:43.729045 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:43.729046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:43.729046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:43.729186 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:43.729264 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:43.729627 (+ 363us) server_negotiation.cc:299] Negotiation successful
0618 23:31:43.729741 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 23:31:43.730086 3016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:43.718033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.1:58077)
0618 23:31:43.718215 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:43.718261 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:43.719224 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:43.719710 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:43.719714 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:43.720073 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:43.720360 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:43.720364 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:43.721213 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:43.721214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:43.722050 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:43.722052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:43.723501 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:43.723502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:43.723693 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:43.724325 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:43.724333 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:43.726098 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:43.728291 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:43.728303 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:43.728306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:43.728604 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:43.728839 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:43.728840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:43.728840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:43.728969 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:43.729265 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:43.729273 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:43.729448 (+ 175us) client_negotiation.cc:769] Sending connection context
0618 23:31:43.729565 (+ 117us) client_negotiation.cc:240] Negotiation successful
0618 23:31:43.729788 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":104,"threads_started":1}
I20260618 23:31:43.730695 3014 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44289
I20260618 23:31:43.730950 3014 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:43.731221 3014 heartbeater.cc:511] Master 127.0.98.62:44289 requested a full tablet report, sending...
I20260618 23:31:43.732141 2823 ts_manager.cc:194] Registered new tserver with Master: 2156571d159643729eb6a8f41a4d7a3d (127.0.98.1:37547)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260618 23:31:43.732894 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260618 23:31:43.748716 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44289
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:43.833841 3019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:43.834064 3019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:43.834090 3019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:43.835278 3019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:43.835330 3019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:43.835357 3019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:31:43.836930 3019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:43.837179 3019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:43.837448 3019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:43.843206 3024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:43.843333 3027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:43.843417 3019 server_base.cc:1034] running on GCE node
W20260618 23:31:43.843206 3025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:43.843850 3019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:43.844184 3019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:43.845402 3019 hybrid_clock.cc:648] HybridClock initialized: now 1781825503845362 us; error 61 us; skew 500 ppm
Jun 18 23:31:43 dist-test-slave-6224 krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:43.848533 3019 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:31:43.849128 3019 webserver.cc:460] Webserver started at http://127.0.98.2:40883/ using document root <none> and password file <none>
I20260618 23:31:43.849457 3019 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:43.849536 3019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:43.849712 3019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:43.850992 3019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "8b8d3c1946894089af4ee4356899bcb0"
format_stamp: "Formatted at 2026-06-18 23:31:43 on dist-test-slave-6224"
server_key: "60596791d56ce8c399d181b00171ca8f"
server_key_iv: "ea8406f29175fb65d2a8bd11d09b0e4b"
server_key_version: "encryptionkey@0"
I20260618 23:31:43.851421 3019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8b8d3c1946894089af4ee4356899bcb0"
format_stamp: "Formatted at 2026-06-18 23:31:43 on dist-test-slave-6224"
server_key: "60596791d56ce8c399d181b00171ca8f"
server_key_iv: "ea8406f29175fb65d2a8bd11d09b0e4b"
server_key_version: "encryptionkey@0"
I20260618 23:31:43.853065 3019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:31:43.854089 3034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:43.854367 3019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:43.854497 3019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "8b8d3c1946894089af4ee4356899bcb0"
format_stamp: "Formatted at 2026-06-18 23:31:43 on dist-test-slave-6224"
server_key: "60596791d56ce8c399d181b00171ca8f"
server_key_iv: "ea8406f29175fb65d2a8bd11d09b0e4b"
server_key_version: "encryptionkey@0"
I20260618 23:31:43.854590 3019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:43.908953 3019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:43.911319 3019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:43.911456 3019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:43.911796 3019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:43.912228 3019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:43.912267 3019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:43.912293 3019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:43.912307 3019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:43.920444 3019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34123
I20260618 23:31:43.920495 3147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34123 every 8 connection(s)
I20260618 23:31:43.920912 3019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:31:43.926084 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 3019
I20260618 23:31:43.926191 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:31:43.926422 392 external_mini_cluster.cc:1428] Setting key 4a734dbbff46c2e9b3fbab9a2b5be0a5
Jun 18 23:31:43 dist-test-slave-6224 krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:43.933090 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:43.921864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50381 (local address 127.0.98.62:44289)
0618 23:31:43.921972 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:43.921975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:43.922232 (+ 257us) server_negotiation.cc:407] Connection header received
0618 23:31:43.923291 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:43.923292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:43.923305 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:43.923387 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:43.924234 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:43.924753 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:43.926022 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:43.926303 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:43.926398 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:43.929644 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:43.929645 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:43.929646 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:43.929654 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:43.931108 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:43.931654 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:43.931656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:43.931656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:43.931781 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:43.932168 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:43.932169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:43.932169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:43.932323 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:43.932404 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:43.932818 (+ 414us) server_negotiation.cc:299] Negotiation successful
0618 23:31:43.932938 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260618 23:31:43.933247 3150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:43.921850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.2:50381)
0618 23:31:43.922053 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:43.922106 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:43.923178 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:43.923455 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:43.923459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:43.923860 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:43.924111 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:43.924116 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:43.924881 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:43.924882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:43.925910 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:43.925914 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:43.926423 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:43.926426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:43.926637 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:43.927573 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:43.927582 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:43.929524 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:43.931214 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:43.931221 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:43.931224 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:43.931569 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:43.931878 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:43.931879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:43.931879 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:43.932032 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:43.932408 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:43.932414 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:43.932607 (+ 193us) client_negotiation.cc:769] Sending connection context
0618 23:31:43.932728 (+ 121us) client_negotiation.cc:240] Negotiation successful
0618 23:31:43.932942 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":115,"threads_started":1}
I20260618 23:31:43.933905 3148 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44289
I20260618 23:31:43.934012 3148 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:43.934214 3148 heartbeater.cc:511] Master 127.0.98.62:44289 requested a full tablet report, sending...
I20260618 23:31:43.934732 2823 ts_manager.cc:194] Registered new tserver with Master: 8b8d3c1946894089af4ee4356899bcb0 (127.0.98.2:34123)
I20260618 23:31:43.935164 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50381
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260618 23:31:43.954658 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44289
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:44.038031 3153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:44.038249 3153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:44.038271 3153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:44.039415 3153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:44.039467 3153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:44.039494 3153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:31:44.041052 3153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:44.041288 3153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:44.041548 3153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:44.046813 3161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:44.046808 3158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:44.046824 3159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:44.047152 3153 server_base.cc:1034] running on GCE node
I20260618 23:31:44.047379 3153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:44.047675 3153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:44.048838 3153 hybrid_clock.cc:648] HybridClock initialized: now 1781825504048821 us; error 37 us; skew 500 ppm
Jun 18 23:31:44 dist-test-slave-6224 krb5kdc[2778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:44.051491 3153 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:31:44.052055 3153 webserver.cc:460] Webserver started at http://127.0.98.3:41535/ using document root <none> and password file <none>
I20260618 23:31:44.052290 3153 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:44.052347 3153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:44.052484 3153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:44.053529 3153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "ebad30401b6a4e8fb781af466267577f"
format_stamp: "Formatted at 2026-06-18 23:31:44 on dist-test-slave-6224"
server_key: "aaa0e10cfed7a919bfdbdd33a070b6b6"
server_key_iv: "e8a3a7668af6469e2b035a917982a55d"
server_key_version: "encryptionkey@0"
I20260618 23:31:44.053930 3153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ebad30401b6a4e8fb781af466267577f"
format_stamp: "Formatted at 2026-06-18 23:31:44 on dist-test-slave-6224"
server_key: "aaa0e10cfed7a919bfdbdd33a070b6b6"
server_key_iv: "e8a3a7668af6469e2b035a917982a55d"
server_key_version: "encryptionkey@0"
I20260618 23:31:44.055384 3153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:31:44.056716 3168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:44.057041 3153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:31:44.057157 3153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "ebad30401b6a4e8fb781af466267577f"
format_stamp: "Formatted at 2026-06-18 23:31:44 on dist-test-slave-6224"
server_key: "aaa0e10cfed7a919bfdbdd33a070b6b6"
server_key_iv: "e8a3a7668af6469e2b035a917982a55d"
server_key_version: "encryptionkey@0"
I20260618 23:31:44.057245 3153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:44.074880 3153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:44.077368 3153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:44.077527 3153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:44.077824 3153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:44.078132 3153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:44.078186 3153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:44.078230 3153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:44.078259 3153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:44.085346 3153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38869
I20260618 23:31:44.085403 3281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38869 every 8 connection(s)
I20260618 23:31:44.085717 3153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:31:44.090955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 3153
I20260618 23:31:44.091082 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:31:44.091274 392 external_mini_cluster.cc:1428] Setting key 808acb26d4fd833395f1f7198a5a9c9c
Jun 18 23:31:44 dist-test-slave-6224 krb5kdc[2778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:44.096383 3284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:44.086522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44289 (local address 127.0.98.3:42089)
0618 23:31:44.086715 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:44.086753 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:44.087831 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:44.088094 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:44.088097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:44.088346 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:44.088537 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:44.088540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:44.089244 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:44.089245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:44.090084 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:44.090086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:44.090489 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:44.090490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:44.090584 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:44.091260 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:44.091270 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:44.093188 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:44.094948 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:44.094953 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:44.094955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:44.095212 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:44.095428 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:44.095429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:44.095429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:44.095522 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:44.095868 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:44.095872 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:44.095988 (+ 116us) client_negotiation.cc:769] Sending connection context
0618 23:31:44.096102 (+ 114us) client_negotiation.cc:240] Negotiation successful
0618 23:31:44.096220 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":100,"threads_started":1}
I20260618 23:31:44.096407 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:44.086562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42089 (local address 127.0.98.62:44289)
0618 23:31:44.086615 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:44.086617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:44.086901 (+ 284us) server_negotiation.cc:407] Connection header received
0618 23:31:44.087944 (+ 1043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:44.087945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:44.087959 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:44.088020 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:44.088617 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:44.089147 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:44.090191 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:44.090401 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:44.090473 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:44.093293 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:44.093294 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:44.093295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:44.093302 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:44.094854 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:44.095307 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:44.095308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:44.095308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:44.095362 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:44.095657 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:44.095658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:44.095658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:44.095781 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:44.095840 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:44.096248 (+ 408us) server_negotiation.cc:299] Negotiation successful
0618 23:31:44.096331 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260618 23:31:44.097074 3282 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44289
I20260618 23:31:44.097183 3282 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:44.097359 3282 heartbeater.cc:511] Master 127.0.98.62:44289 requested a full tablet report, sending...
I20260618 23:31:44.097790 2823 ts_manager.cc:194] Registered new tserver with Master: ebad30401b6a4e8fb781af466267577f (127.0.98.3:38869)
I20260618 23:31:44.098205 2823 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42089
I20260618 23:31:44.103590 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260618 23:31:44.116752 2863 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:31:44.115668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48600 (local address 127.0.98.62:44289)
0618 23:31:44.115762 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:44.115765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:44.115882 (+ 117us) server_negotiation.cc:407] Connection header received
0618 23:31:44.115972 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:44.115973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:44.115985 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:44.116032 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:44.116672 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48600: BlockingRecv error: recv got EOF from 127.0.0.1:48600 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 23:31:44.117758 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 2885
I20260618 23:31:44.122843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 3019
I20260618 23:31:44.128088 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 3153
I20260618 23:31:44.133543 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 2791
2026-06-18T23:31:44Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2005 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:31:44 dist-test-slave-6224 krb5kdc[3294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:31:44 dist-test-slave-6224 krb5kdc[3294](info): set up 2 sockets
Jun 18 23:31:44 dist-test-slave-6224 krb5kdc[3294](info): commencing operation
krb5kdc: starting...
W20260618 23:31:45.187971 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:31:45 dist-test-slave-6224 krb5kdc[3294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:31:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:31:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:31:45.318248 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33735
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:45.402742 3307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:45.403026 3307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:45.403067 3307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:45.404327 3307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:31:45.404397 3307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:45.404420 3307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:45.404440 3307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:31:45.404457 3307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:31:45.405929 3307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:45.406216 3307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:45.406445 3307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:45.411684 3312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:45.411751 3313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:45.411676 3315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:45.411835 3307 server_base.cc:1034] running on GCE node
I20260618 23:31:45.412254 3307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:45.412560 3307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:45.413713 3307 hybrid_clock.cc:648] HybridClock initialized: now 1781825505413690 us; error 54 us; skew 500 ppm
Jun 18 23:31:45 dist-test-slave-6224 krb5kdc[3294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:45.416363 3307 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:31:45.416879 3307 webserver.cc:460] Webserver started at http://127.0.98.62:35229/ using document root <none> and password file <none>
I20260618 23:31:45.417114 3307 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:45.417163 3307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:45.417320 3307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:45.418423 3307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "71645da0e23d478c9309acf9e7af10ec"
format_stamp: "Formatted at 2026-06-18 23:31:45 on dist-test-slave-6224"
server_key: "a78f56727339658395d05bda86de3caa"
server_key_iv: "e1ed239e83094681544d131f1aa43897"
server_key_version: "encryptionkey@0"
I20260618 23:31:45.418790 3307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "71645da0e23d478c9309acf9e7af10ec"
format_stamp: "Formatted at 2026-06-18 23:31:45 on dist-test-slave-6224"
server_key: "a78f56727339658395d05bda86de3caa"
server_key_iv: "e1ed239e83094681544d131f1aa43897"
server_key_version: "encryptionkey@0"
I20260618 23:31:45.420332 3307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:31:45.421195 3322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:45.421443 3307 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:45.421551 3307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "71645da0e23d478c9309acf9e7af10ec"
format_stamp: "Formatted at 2026-06-18 23:31:45 on dist-test-slave-6224"
server_key: "a78f56727339658395d05bda86de3caa"
server_key_iv: "e1ed239e83094681544d131f1aa43897"
server_key_version: "encryptionkey@0"
I20260618 23:31:45.421638 3307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:45.475811 3307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:45.478430 3307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:45.478565 3307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:45.483778 3307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33735
I20260618 23:31:45.483975 3374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33735 every 8 connection(s)
I20260618 23:31:45.484274 3307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:31:45.485109 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 3307
I20260618 23:31:45.485206 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:31:45.485437 392 external_mini_cluster.cc:1428] Setting key 8da57c5859134fa9bffa71f0acf41680
I20260618 23:31:45.485431 3375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:45.489210 3375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec: Bootstrap starting.
I20260618 23:31:45.489984 3375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:45.490288 3375 log.cc:824] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:45.491083 3375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec: No bootstrap required, opened a new log
Jun 18 23:31:45 dist-test-slave-6224 krb5kdc[3294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:45.493494 3375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71645da0e23d478c9309acf9e7af10ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33735 } }
I20260618 23:31:45.493664 3375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:45.493727 3375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71645da0e23d478c9309acf9e7af10ec, State: Initialized, Role: FOLLOWER
I20260618 23:31:45.493893 3375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71645da0e23d478c9309acf9e7af10ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33735 } }
I20260618 23:31:45.493988 3375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:31:45.494030 3375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:31:45.494078 3375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:45.494735 3375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71645da0e23d478c9309acf9e7af10ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33735 } }
I20260618 23:31:45.494877 3375 leader_election.cc:302] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71645da0e23d478c9309acf9e7af10ec; no voters:
I20260618 23:31:45.495108 3375 leader_election.cc:288] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:31:45.495160 3380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:45.495277 3380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec [term 1 LEADER]: Becoming Leader. State: Replica: 71645da0e23d478c9309acf9e7af10ec, State: Running, Role: LEADER
I20260618 23:31:45.495395 3375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:31:45.495472 3380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71645da0e23d478c9309acf9e7af10ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33735 } }
I20260618 23:31:45.496057 3382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71645da0e23d478c9309acf9e7af10ec. Latest consensus state: current_term: 1 leader_uuid: "71645da0e23d478c9309acf9e7af10ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71645da0e23d478c9309acf9e7af10ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33735 } } }
I20260618 23:31:45.496166 3382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:45.496457 3381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71645da0e23d478c9309acf9e7af10ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71645da0e23d478c9309acf9e7af10ec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33735 } } }
I20260618 23:31:45.496527 3381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:45.496948 3389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:31:45.497515 3389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:31:45.498309 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:45.486122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52624 (local address 127.0.98.62:33735)
0618 23:31:45.486341 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:45.486349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:45.486380 (+ 31us) server_negotiation.cc:407] Connection header received
0618 23:31:45.486888 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:45.486895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:45.486958 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:45.487126 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:45.488039 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:45.489047 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:45.490204 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:45.490745 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:45.490854 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:45.493842 (+ 2988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:45.493848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:45.493849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:45.493864 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:45.496170 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:45.496639 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:45.496642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:45.496643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:45.496807 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:45.497063 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:45.497064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:45.497064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:45.497498 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:45.497659 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:45.497901 (+ 242us) server_negotiation.cc:299] Negotiation successful
0618 23:31:45.498101 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":115,"threads_started":1}
I20260618 23:31:45.499586 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:31:45.499650 3389 catalog_manager.cc:1347] Generated new cluster ID: bd122971c8884ac681b08519303acf2c
I20260618 23:31:45.499696 3389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:31:45.505955 3389 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:31:45.506819 3389 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:31:45.517158 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:45.512238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52634 (local address 127.0.98.62:33735)
0618 23:31:45.512290 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:45.512292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:45.512410 (+ 118us) server_negotiation.cc:407] Connection header received
0618 23:31:45.512478 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:45.512479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:45.512493 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:45.512596 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:45.513101 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:45.513923 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:45.515096 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:45.515518 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:45.515641 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:45.516488 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:45.516490 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:45.516490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:31:45.516808 (+ 318us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:45.516831 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:31:45.516839 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:45.516908 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:45.517020 (+ 112us) server_negotiation.cc:299] Negotiation successful
0618 23:31:45.517056 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260618 23:31:45.518499 3389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71645da0e23d478c9309acf9e7af10ec: Generated new TSK 0
I20260618 23:31:45.518684 3389 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:31:45.521323 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 3307
2026-06-18T23:31:45Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1384 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:31:45 dist-test-slave-6224 krb5kdc[3408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:31:45 dist-test-slave-6224 krb5kdc[3408](info): set up 2 sockets
Jun 18 23:31:45 dist-test-slave-6224 krb5kdc[3408](info): commencing operation
krb5kdc: starting...
W20260618 23:31:46.595646 392 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/test-user.keytab.
W20260618 23:31:47.406620 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.689s user 0.000s sys 0.002s
Jun 18 23:31:47 dist-test-slave-6224 krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:31:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:31:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260618 23:31:48.383931 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.951s user 0.000s sys 0.004s
I20260618 23:31:48.384394 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33703
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:48.473743 3421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:48.474043 3421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:48.474084 3421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:48.475452 3421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:31:48.475531 3421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:48.475557 3421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:48.475600 3421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:31:48.475628 3421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:31:48.477317 3421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:48.477628 3421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:48.477892 3421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:48.483403 3426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:48.483388 3427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:48.483415 3429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:48.483680 3421 server_base.cc:1034] running on GCE node
I20260618 23:31:48.484056 3421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:48.484409 3421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:48.485574 3421 hybrid_clock.cc:648] HybridClock initialized: now 1781825508485550 us; error 40 us; skew 500 ppm
Jun 18 23:31:48 dist-test-slave-6224 krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:48.493577 3421 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:31:48.494206 3421 webserver.cc:460] Webserver started at http://127.0.98.62:33143/ using document root <none> and password file <none>
I20260618 23:31:48.494472 3421 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:48.494522 3421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:48.494719 3421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:48.496028 3421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "bc565fecd5af4a0fb89bbeed1cb78847"
format_stamp: "Formatted at 2026-06-18 23:31:48 on dist-test-slave-6224"
server_key: "aabed114844c5dffe068112c846cca82"
server_key_iv: "a551086fe4c9baba1b38236c5953aa1a"
server_key_version: "encryptionkey@0"
I20260618 23:31:48.496439 3421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "bc565fecd5af4a0fb89bbeed1cb78847"
format_stamp: "Formatted at 2026-06-18 23:31:48 on dist-test-slave-6224"
server_key: "aabed114844c5dffe068112c846cca82"
server_key_iv: "a551086fe4c9baba1b38236c5953aa1a"
server_key_version: "encryptionkey@0"
I20260618 23:31:48.497980 3421 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:31:48.498934 3436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:48.499228 3421 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:48.499353 3421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "bc565fecd5af4a0fb89bbeed1cb78847"
format_stamp: "Formatted at 2026-06-18 23:31:48 on dist-test-slave-6224"
server_key: "aabed114844c5dffe068112c846cca82"
server_key_iv: "a551086fe4c9baba1b38236c5953aa1a"
server_key_version: "encryptionkey@0"
I20260618 23:31:48.499434 3421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:48.530835 3421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:48.533170 3421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:48.533360 3421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:48.538820 3421 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33703
I20260618 23:31:48.538985 3488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33703 every 8 connection(s)
I20260618 23:31:48.539341 3421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:31:48.540433 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 3421
I20260618 23:31:48.540549 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:31:48.540715 3489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:48.540782 392 external_mini_cluster.cc:1428] Setting key 8094fb3eae6677d5ca423b06ae46e0a8
I20260618 23:31:48.544457 3489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847: Bootstrap starting.
I20260618 23:31:48.545125 3489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:48.545410 3489 log.cc:824] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:48.546131 3489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847: No bootstrap required, opened a new log
Jun 18 23:31:48 dist-test-slave-6224 krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:48.548542 3489 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc565fecd5af4a0fb89bbeed1cb78847" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33703 } }
I20260618 23:31:48.548713 3489 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:48.548738 3489 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc565fecd5af4a0fb89bbeed1cb78847, State: Initialized, Role: FOLLOWER
I20260618 23:31:48.548911 3489 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc565fecd5af4a0fb89bbeed1cb78847" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33703 } }
I20260618 23:31:48.548996 3489 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:31:48.549019 3489 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:31:48.549141 3489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:48.549984 3489 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc565fecd5af4a0fb89bbeed1cb78847" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33703 } }
I20260618 23:31:48.550091 3489 leader_election.cc:302] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc565fecd5af4a0fb89bbeed1cb78847; no voters:
I20260618 23:31:48.550311 3489 leader_election.cc:288] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:31:48.550379 3494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:48.550581 3494 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847 [term 1 LEADER]: Becoming Leader. State: Replica: bc565fecd5af4a0fb89bbeed1cb78847, State: Running, Role: LEADER
I20260618 23:31:48.550635 3489 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:31:48.550794 3494 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc565fecd5af4a0fb89bbeed1cb78847" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33703 } }
I20260618 23:31:48.551268 3494 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc565fecd5af4a0fb89bbeed1cb78847. Latest consensus state: current_term: 1 leader_uuid: "bc565fecd5af4a0fb89bbeed1cb78847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc565fecd5af4a0fb89bbeed1cb78847" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33703 } } }
I20260618 23:31:48.551270 3495 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc565fecd5af4a0fb89bbeed1cb78847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc565fecd5af4a0fb89bbeed1cb78847" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33703 } } }
I20260618 23:31:48.551508 3495 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847 [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:48.551787 3494 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847 [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:48.551990 3503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:31:48.552673 3503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:31:48.553802 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:48.541485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55588 (local address 127.0.98.62:33703)
0618 23:31:48.541716 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:48.541725 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:48.541762 (+ 37us) server_negotiation.cc:407] Connection header received
0618 23:31:48.542505 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:48.542513 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:48.542569 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:48.542771 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:48.543469 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:48.544538 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:48.545435 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:48.545980 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:48.546125 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:48.549155 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:48.549160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:48.549161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:48.549184 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:48.551474 (+ 2290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:48.551992 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:48.551995 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:48.551996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:48.552100 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:48.552318 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:48.552319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:48.552319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:48.552659 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:48.553350 (+ 691us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:48.553428 (+ 78us) server_negotiation.cc:299] Negotiation successful
0618 23:31:48.553578 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":110,"threads_started":1}
I20260618 23:31:48.554700 3503 catalog_manager.cc:1347] Generated new cluster ID: d72c7969595043338c804e7aeedf69d0
I20260618 23:31:48.554754 3503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:31:48.560731 3503 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:31:48.561285 3503 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:31:48.573441 3503 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc565fecd5af4a0fb89bbeed1cb78847: Generated new TSK 0
I20260618 23:31:48.573726 3503 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260618 23:31:48.604539 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33703
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:48.693560 3515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:48.693781 3515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:48.693805 3515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:48.695073 3515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:48.695122 3515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:48.695148 3515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:31:48.696621 3515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:48.696841 3515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:48.697036 3515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:48.702622 3523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:48.702638 3520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:48.702631 3521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:48.703163 3515 server_base.cc:1034] running on GCE node
I20260618 23:31:48.703385 3515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:48.703763 3515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:48.704947 3515 hybrid_clock.cc:648] HybridClock initialized: now 1781825508704937 us; error 47 us; skew 500 ppm
Jun 18 23:31:48 dist-test-slave-6224 krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:48.707973 3515 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:31:48.708516 3515 webserver.cc:460] Webserver started at http://127.0.98.1:46249/ using document root <none> and password file <none>
I20260618 23:31:48.708773 3515 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:48.708844 3515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:48.709007 3515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:48.710227 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "8d7dfaf6da2f4b868a5cebe85e7a5121"
format_stamp: "Formatted at 2026-06-18 23:31:48 on dist-test-slave-6224"
server_key: "bf4703b54d4bc57336baa87d19ac6873"
server_key_iv: "af404becffc1c0984cc8dc5cfed8a34a"
server_key_version: "encryptionkey@0"
I20260618 23:31:48.710631 3515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8d7dfaf6da2f4b868a5cebe85e7a5121"
format_stamp: "Formatted at 2026-06-18 23:31:48 on dist-test-slave-6224"
server_key: "bf4703b54d4bc57336baa87d19ac6873"
server_key_iv: "af404becffc1c0984cc8dc5cfed8a34a"
server_key_version: "encryptionkey@0"
I20260618 23:31:48.712203 3515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:31:48.713057 3530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:48.713264 3515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:31:48.713373 3515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "8d7dfaf6da2f4b868a5cebe85e7a5121"
format_stamp: "Formatted at 2026-06-18 23:31:48 on dist-test-slave-6224"
server_key: "bf4703b54d4bc57336baa87d19ac6873"
server_key_iv: "af404becffc1c0984cc8dc5cfed8a34a"
server_key_version: "encryptionkey@0"
I20260618 23:31:48.713451 3515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:48.728652 3515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:48.730943 3515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:48.731153 3515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:48.731436 3515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:48.731848 3515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:48.731906 3515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:48.731946 3515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:48.731995 3515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:48.739276 3515 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39821
I20260618 23:31:48.739326 3643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39821 every 8 connection(s)
I20260618 23:31:48.739717 3515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:31:48.740399 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 3515
I20260618 23:31:48.740537 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:31:48.740767 392 external_mini_cluster.cc:1428] Setting key 956d299f6761ef591c90825733864259
Jun 18 23:31:48 dist-test-slave-6224 krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260618 23:31:48.751638 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:48.740714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53393 (local address 127.0.98.62:33703)
0618 23:31:48.740779 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:48.740782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:48.740982 (+ 200us) server_negotiation.cc:407] Connection header received
0618 23:31:48.741719 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:48.741720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:48.741733 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:48.741876 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:48.742997 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:48.743513 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:48.744843 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:48.745226 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:48.745319 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:48.748093 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:48.748095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:48.748095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:48.748104 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:48.749894 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:48.750389 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:48.750390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:48.750390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:48.750451 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:48.750766 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:48.750768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:48.750769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:48.751005 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:48.751075 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:48.751386 (+ 311us) server_negotiation.cc:299] Negotiation successful
0618 23:31:48.751492 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260618 23:31:48.751888 3646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:48.740654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33703 (local address 127.0.98.1:53393)
0618 23:31:48.740830 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:48.740876 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:48.741592 (+ 716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:48.741935 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:48.741939 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:48.742554 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:48.742871 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:48.742879 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:48.743667 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:48.743668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:48.744732 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:48.744734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:48.745342 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:48.745343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:48.745473 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:48.746121 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:48.746131 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:48.747979 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:48.749993 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:48.749999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:48.750001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:48.750305 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:48.750523 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:48.750523 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:48.750524 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:48.750647 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:48.751074 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:48.751082 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:48.751243 (+ 161us) client_negotiation.cc:769] Sending connection context
0618 23:31:48.751363 (+ 120us) client_negotiation.cc:240] Negotiation successful
0618 23:31:48.751533 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260618 23:31:48.752693 3644 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33703
I20260618 23:31:48.752806 3644 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:48.753012 3644 heartbeater.cc:511] Master 127.0.98.62:33703 requested a full tablet report, sending...
I20260618 23:31:48.753669 3453 ts_manager.cc:194] Registered new tserver with Master: 8d7dfaf6da2f4b868a5cebe85e7a5121 (127.0.98.1:39821)
I20260618 23:31:48.754377 3453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260618 23:31:48.770612 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33703
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:48.856665 3649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:48.856915 3649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:48.856942 3649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:48.858222 3649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:48.858275 3649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:48.858302 3649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:31:48.859880 3649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:48.860126 3649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:48.860383 3649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:31:48.866011 3649 server_base.cc:1034] running on GCE node
W20260618 23:31:48.865958 3657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:48.865976 3655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:48.865942 3654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:48.866535 3649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:48.866844 3649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:48.868055 3649 hybrid_clock.cc:648] HybridClock initialized: now 1781825508868021 us; error 55 us; skew 500 ppm
Jun 18 23:31:48 dist-test-slave-6224 krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:48.870922 3649 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:31:48.871452 3649 webserver.cc:460] Webserver started at http://127.0.98.2:39241/ using document root <none> and password file <none>
I20260618 23:31:48.871744 3649 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:48.871819 3649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:48.871976 3649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:48.873215 3649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "cdf23bfe1ba0478295244ce642fa4b5b"
format_stamp: "Formatted at 2026-06-18 23:31:48 on dist-test-slave-6224"
server_key: "725154acf8a80dd1c08ed1680a3e0821"
server_key_iv: "56bfa3227628bc916917e389b3a56ce0"
server_key_version: "encryptionkey@0"
I20260618 23:31:48.873631 3649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cdf23bfe1ba0478295244ce642fa4b5b"
format_stamp: "Formatted at 2026-06-18 23:31:48 on dist-test-slave-6224"
server_key: "725154acf8a80dd1c08ed1680a3e0821"
server_key_iv: "56bfa3227628bc916917e389b3a56ce0"
server_key_version: "encryptionkey@0"
I20260618 23:31:48.875227 3649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:31:48.876259 3664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:48.876545 3649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260618 23:31:48.876667 3649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "cdf23bfe1ba0478295244ce642fa4b5b"
format_stamp: "Formatted at 2026-06-18 23:31:48 on dist-test-slave-6224"
server_key: "725154acf8a80dd1c08ed1680a3e0821"
server_key_iv: "56bfa3227628bc916917e389b3a56ce0"
server_key_version: "encryptionkey@0"
I20260618 23:31:48.876756 3649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:48.890393 3649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:48.893170 3649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:48.893375 3649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:48.893685 3649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:48.894060 3649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:48.894117 3649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:48.894163 3649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:48.894192 3649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:48.901952 3649 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41245
I20260618 23:31:48.902016 3777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41245 every 8 connection(s)
I20260618 23:31:48.902410 3649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:31:48.906610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 3649
I20260618 23:31:48.906740 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:31:48.906963 392 external_mini_cluster.cc:1428] Setting key 587b7e86d28227fbeaa4fb422014220b
Jun 18 23:31:48 dist-test-slave-6224 krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825508, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:48.913918 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:48.903303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37351 (local address 127.0.98.62:33703)
0618 23:31:48.903384 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:48.903386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:48.903674 (+ 288us) server_negotiation.cc:407] Connection header received
0618 23:31:48.904335 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:48.904337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:48.904352 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:48.904414 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:48.905510 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:48.906070 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:48.907634 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:48.907912 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:48.907989 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:48.910803 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:48.910805 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:48.910806 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:48.910822 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:48.912357 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:48.912818 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:48.912820 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:48.912820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:48.912882 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:48.913140 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:48.913141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:48.913141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:48.913302 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:48.913411 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:48.913713 (+ 302us) server_negotiation.cc:299] Negotiation successful
0618 23:31:48.913805 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 23:31:48.914144 3780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:48.903263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33703 (local address 127.0.98.2:37351)
0618 23:31:48.903496 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:48.903541 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:48.904203 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:48.904505 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:48.904510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:48.905152 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:48.905399 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:48.905404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:48.906175 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:48.906176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:48.907490 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:48.907493 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:48.908058 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:48.908061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:48.908281 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:48.908979 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:48.908991 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:48.910678 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:48.912468 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:48.912475 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:48.912478 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:48.912725 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:48.912951 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:48.912952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:48.912952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:48.913072 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:48.913394 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:48.913409 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:48.913568 (+ 159us) client_negotiation.cc:769] Sending connection context
0618 23:31:48.913691 (+ 123us) client_negotiation.cc:240] Negotiation successful
0618 23:31:48.913890 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":123,"threads_started":1}
I20260618 23:31:48.914615 3778 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33703
I20260618 23:31:48.914784 3778 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:48.915066 3778 heartbeater.cc:511] Master 127.0.98.62:33703 requested a full tablet report, sending...
I20260618 23:31:48.915597 3453 ts_manager.cc:194] Registered new tserver with Master: cdf23bfe1ba0478295244ce642fa4b5b (127.0.98.2:41245)
I20260618 23:31:48.916030 3453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37351
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260618 23:31:48.934597 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33703
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:49.026399 3783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:49.026662 3783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:49.026691 3783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:49.028060 3783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:49.028117 3783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:49.028148 3783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:31:49.029731 3783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:49.029966 3783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:49.030280 3783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:49.035925 3789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:49.035926 3791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:49.036008 3788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:49.036232 3783 server_base.cc:1034] running on GCE node
I20260618 23:31:49.036429 3783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:49.036664 3783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:49.037840 3783 hybrid_clock.cc:648] HybridClock initialized: now 1781825509037821 us; error 51 us; skew 500 ppm
Jun 18 23:31:49 dist-test-slave-6224 krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:49.040967 3783 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:31:49.041497 3783 webserver.cc:460] Webserver started at http://127.0.98.3:39299/ using document root <none> and password file <none>
I20260618 23:31:49.041800 3783 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:49.041878 3783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:49.042042 3783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:49.043298 3783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "f7501063d2434b57b5ab9abb46b1a893"
format_stamp: "Formatted at 2026-06-18 23:31:49 on dist-test-slave-6224"
server_key: "77a03b8bba68f842ee61b423419f9e3f"
server_key_iv: "f913e38471228a5c9197b88e7830fdd5"
server_key_version: "encryptionkey@0"
I20260618 23:31:49.043869 3783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f7501063d2434b57b5ab9abb46b1a893"
format_stamp: "Formatted at 2026-06-18 23:31:49 on dist-test-slave-6224"
server_key: "77a03b8bba68f842ee61b423419f9e3f"
server_key_iv: "f913e38471228a5c9197b88e7830fdd5"
server_key_version: "encryptionkey@0"
I20260618 23:31:49.045624 3783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:31:49.046669 3798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:49.046972 3783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:49.047096 3783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "f7501063d2434b57b5ab9abb46b1a893"
format_stamp: "Formatted at 2026-06-18 23:31:49 on dist-test-slave-6224"
server_key: "77a03b8bba68f842ee61b423419f9e3f"
server_key_iv: "f913e38471228a5c9197b88e7830fdd5"
server_key_version: "encryptionkey@0"
I20260618 23:31:49.047191 3783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:49.092763 3783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:49.095603 3783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:49.095798 3783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:49.096098 3783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:49.096482 3783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:49.096539 3783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:49.096583 3783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:49.096627 3783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:49.104593 3783 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34811
I20260618 23:31:49.104671 3911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34811 every 8 connection(s)
I20260618 23:31:49.105082 3783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:31:49.111716 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 3783
I20260618 23:31:49.111865 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:31:49.112080 392 external_mini_cluster.cc:1428] Setting key 5d8a11a19042d268c44b9e096bb5b415
Jun 18 23:31:49 dist-test-slave-6224 krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825509, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:49.116315 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:49.106037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34453 (local address 127.0.98.62:33703)
0618 23:31:49.106127 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:49.106129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:49.106432 (+ 303us) server_negotiation.cc:407] Connection header received
0618 23:31:49.107092 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:49.107094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:49.107109 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:49.107173 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:49.108377 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:49.108944 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:49.110092 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:49.110316 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:49.110412 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:49.113222 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:49.113223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:49.113224 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:49.113233 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:49.114673 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:49.115129 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:49.115130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:49.115131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:49.115192 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:49.115471 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:49.115472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:49.115472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:49.115675 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:49.115782 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:49.116114 (+ 332us) server_negotiation.cc:299] Negotiation successful
0618 23:31:49.116210 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 23:31:49.116549 3914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:49.106020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33703 (local address 127.0.98.3:34453)
0618 23:31:49.106254 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:49.106312 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:49.106969 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:49.107273 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:49.107278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:49.108014 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:49.108259 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:49.108264 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:49.109049 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:49.109071 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:49.109991 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:49.109993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:49.110420 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:49.110421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:49.110564 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:49.111233 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:49.111243 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:49.113117 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:49.114778 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:49.114786 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:49.114789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:49.115035 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:49.115263 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:49.115264 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:49.115264 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:49.115391 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:49.115742 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:49.115749 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:49.115943 (+ 194us) client_negotiation.cc:769] Sending connection context
0618 23:31:49.116084 (+ 141us) client_negotiation.cc:240] Negotiation successful
0618 23:31:49.116281 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":131,"threads_started":1}
I20260618 23:31:49.117179 3912 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33703
I20260618 23:31:49.117313 3912 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:49.117503 3912 heartbeater.cc:511] Master 127.0.98.62:33703 requested a full tablet report, sending...
I20260618 23:31:49.117901 3453 ts_manager.cc:194] Registered new tserver with Master: f7501063d2434b57b5ab9abb46b1a893 (127.0.98.3:34811)
I20260618 23:31:49.118268 3453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34453
I20260618 23:31:49.124290 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 23:31:49 dist-test-slave-6224 krb5kdc[3408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825509, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 18 23:31:49 dist-test-slave-6224 krb5kdc[3408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825509, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:49.149724 3492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:49.141678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55590 (local address 127.0.98.62:33703)
0618 23:31:49.141755 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:49.141757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:49.141771 (+ 14us) server_negotiation.cc:407] Connection header received
0618 23:31:49.141810 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:49.141810 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:49.141822 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:49.141914 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:49.142502 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:49.143037 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:49.144006 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:49.144215 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:49.144278 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:49.146858 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:49.146861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:49.146861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:49.146870 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:49.148355 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:49.148748 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:49.148749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:49.148749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:49.148814 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:49.149105 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:49.149106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:49.149106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:49.149296 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:49.149405 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:49.149557 (+ 152us) server_negotiation.cc:299] Negotiation successful
0618 23:31:49.149646 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260618 23:31:49.150051 3453 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55590
I20260618 23:31:49.150691 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 3515
I20260618 23:31:49.156553 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 3649
I20260618 23:31:49.161936 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 3783
I20260618 23:31:49.167057 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 3421
2026-06-18T23:31:49Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3647 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:31:49 dist-test-slave-6224 krb5kdc[3924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:31:49 dist-test-slave-6224 krb5kdc[3924](info): set up 2 sockets
Jun 18 23:31:49 dist-test-slave-6224 krb5kdc[3924](info): commencing operation
krb5kdc: starting...
W20260618 23:31:50.211493 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:31:50 dist-test-slave-6224 krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:31:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:31:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:31:50.341013 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32913
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:50.424516 3937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:50.424790 3937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:50.424823 3937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:50.425984 3937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:31:50.426061 3937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:50.426087 3937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:50.426116 3937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:31:50.426146 3937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:31:50.427623 3937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:50.427915 3937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:50.428145 3937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:50.433249 3943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:50.433297 3945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:50.433297 3942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:50.433743 3937 server_base.cc:1034] running on GCE node
I20260618 23:31:50.434005 3937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:50.434298 3937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:50.435487 3937 hybrid_clock.cc:648] HybridClock initialized: now 1781825510435467 us; error 46 us; skew 500 ppm
Jun 18 23:31:50 dist-test-slave-6224 krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:50.438247 3937 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:31:50.438776 3937 webserver.cc:460] Webserver started at http://127.0.98.62:44039/ using document root <none> and password file <none>
I20260618 23:31:50.439031 3937 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:50.439082 3937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:50.439230 3937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:50.440415 3937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "741c04641d774365b262d221e087bb9f"
format_stamp: "Formatted at 2026-06-18 23:31:50 on dist-test-slave-6224"
server_key: "8caefec9f5e2d4f628a88ed7d2705fc7"
server_key_iv: "20d600be53a9135586f327b08e243af6"
server_key_version: "encryptionkey@0"
I20260618 23:31:50.440783 3937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "741c04641d774365b262d221e087bb9f"
format_stamp: "Formatted at 2026-06-18 23:31:50 on dist-test-slave-6224"
server_key: "8caefec9f5e2d4f628a88ed7d2705fc7"
server_key_iv: "20d600be53a9135586f327b08e243af6"
server_key_version: "encryptionkey@0"
I20260618 23:31:50.442265 3937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260618 23:31:50.443183 3952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:50.443435 3937 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:50.443543 3937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "741c04641d774365b262d221e087bb9f"
format_stamp: "Formatted at 2026-06-18 23:31:50 on dist-test-slave-6224"
server_key: "8caefec9f5e2d4f628a88ed7d2705fc7"
server_key_iv: "20d600be53a9135586f327b08e243af6"
server_key_version: "encryptionkey@0"
I20260618 23:31:50.443657 3937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:50.455722 3937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:50.457736 3937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:50.457911 3937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:50.462570 3937 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32913
I20260618 23:31:50.462716 4004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32913 every 8 connection(s)
I20260618 23:31:50.463024 3937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:31:50.464095 4005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:50.466790 4005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f: Bootstrap starting.
I20260618 23:31:50.467101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 3937
I20260618 23:31:50.467269 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:31:50.467480 392 external_mini_cluster.cc:1428] Setting key a684d4e3dfc8fedc0282a4fdf85a75ed
I20260618 23:31:50.467633 4005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:50.468004 4005 log.cc:824] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:50.468875 4005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f: No bootstrap required, opened a new log
I20260618 23:31:50.471015 4005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741c04641d774365b262d221e087bb9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32913 } }
I20260618 23:31:50.471189 4005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:50.471252 4005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741c04641d774365b262d221e087bb9f, State: Initialized, Role: FOLLOWER
I20260618 23:31:50.471421 4005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741c04641d774365b262d221e087bb9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32913 } }
I20260618 23:31:50.471518 4005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:31:50.471585 4005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:31:50.471635 4005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:50.472470 4005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741c04641d774365b262d221e087bb9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32913 } }
I20260618 23:31:50.472586 4005 leader_election.cc:302] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 741c04641d774365b262d221e087bb9f; no voters:
I20260618 23:31:50.472750 4005 leader_election.cc:288] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:31:50.472821 4010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:50.473070 4005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:31:50.473464 4010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f [term 1 LEADER]: Becoming Leader. State: Replica: 741c04641d774365b262d221e087bb9f, State: Running, Role: LEADER
I20260618 23:31:50.473582 4010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741c04641d774365b262d221e087bb9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32913 } }
I20260618 23:31:50.474247 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "741c04641d774365b262d221e087bb9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741c04641d774365b262d221e087bb9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32913 } } }
I20260618 23:31:50.474347 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:50.474673 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 741c04641d774365b262d221e087bb9f. Latest consensus state: current_term: 1 leader_uuid: "741c04641d774365b262d221e087bb9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "741c04641d774365b262d221e087bb9f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32913 } } }
I20260618 23:31:50.474754 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f [sys.catalog]: This master's current role is: LEADER
Jun 18 23:31:50 dist-test-slave-6224 krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825510, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:50.474913 4023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:31:50.476003 4023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:31:50.477792 4023 catalog_manager.cc:1347] Generated new cluster ID: 7c03668bbe934e3c876685581adce5db
I20260618 23:31:50.477850 4023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:31:50.480348 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:50.468154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49632 (local address 127.0.98.62:32913)
0618 23:31:50.468363 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:50.468373 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:50.468412 (+ 39us) server_negotiation.cc:407] Connection header received
0618 23:31:50.468904 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:50.468910 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:50.469006 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:50.469179 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:50.469855 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:50.470803 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:50.471781 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:50.472410 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:50.472603 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:50.475892 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:50.475896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:50.475896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:50.475908 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:50.478211 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:50.478703 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:50.478705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:50.478706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:50.478806 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:50.479025 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:50.479026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:50.479026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:50.479344 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:50.479485 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:50.479989 (+ 504us) server_negotiation.cc:299] Negotiation successful
0618 23:31:50.480133 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":101,"threads_started":1}
I20260618 23:31:50.500022 4023 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:31:50.500649 4023 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:31:50.506569 4023 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 741c04641d774365b262d221e087bb9f: Generated new TSK 0
I20260618 23:31:50.506810 4023 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260618 23:31:50.539537 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32913
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:50.629530 4031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:50.629781 4031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:50.629808 4031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:50.631094 4031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:50.631146 4031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:50.631177 4031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:31:50.632815 4031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:50.633108 4031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:50.633394 4031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:50.639153 4036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:50.639148 4037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:50.639194 4031 server_base.cc:1034] running on GCE node
W20260618 23:31:50.639194 4039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:50.639717 4031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:50.640017 4031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:50.641216 4031 hybrid_clock.cc:648] HybridClock initialized: now 1781825510641195 us; error 39 us; skew 500 ppm
Jun 18 23:31:50 dist-test-slave-6224 krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:50.644268 4031 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:31:50.644757 4031 webserver.cc:460] Webserver started at http://127.0.98.1:44529/ using document root <none> and password file <none>
I20260618 23:31:50.645057 4031 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:50.645112 4031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:50.645263 4031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:50.646574 4031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "f671c513452248fbb0b70a03aef0661f"
format_stamp: "Formatted at 2026-06-18 23:31:50 on dist-test-slave-6224"
server_key: "a3f98ccd49e704b2547f1e69aadf49c5"
server_key_iv: "185f9737ace710d1daa3212bbf04af00"
server_key_version: "encryptionkey@0"
I20260618 23:31:50.647017 4031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f671c513452248fbb0b70a03aef0661f"
format_stamp: "Formatted at 2026-06-18 23:31:50 on dist-test-slave-6224"
server_key: "a3f98ccd49e704b2547f1e69aadf49c5"
server_key_iv: "185f9737ace710d1daa3212bbf04af00"
server_key_version: "encryptionkey@0"
I20260618 23:31:50.648715 4031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260618 23:31:50.649624 4046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:50.649888 4031 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:31:50.649971 4031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "f671c513452248fbb0b70a03aef0661f"
format_stamp: "Formatted at 2026-06-18 23:31:50 on dist-test-slave-6224"
server_key: "a3f98ccd49e704b2547f1e69aadf49c5"
server_key_iv: "185f9737ace710d1daa3212bbf04af00"
server_key_version: "encryptionkey@0"
I20260618 23:31:50.650075 4031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:50.663669 4031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:50.666373 4031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:50.666550 4031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:50.666839 4031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:50.667212 4031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:50.667269 4031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:50.667317 4031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:50.667344 4031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:50.674952 4031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42775
I20260618 23:31:50.675016 4159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42775 every 8 connection(s)
I20260618 23:31:50.675432 4031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
Jun 18 23:31:50 dist-test-slave-6224 krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:50.685575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 4031
I20260618 23:31:50.685708 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:31:50.685812 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:50.676384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38875 (local address 127.0.98.62:32913)
0618 23:31:50.676450 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:50.676453 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:50.676726 (+ 273us) server_negotiation.cc:407] Connection header received
0618 23:31:50.677864 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:50.677866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:50.677878 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:50.677928 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:50.678624 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:50.679149 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:50.680260 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:50.680478 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:50.680566 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:50.682980 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:50.682981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:50.682981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:50.682988 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:50.684338 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:50.684782 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:50.684783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:50.684783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:50.684841 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:50.685091 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:50.685091 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:50.685091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:50.685198 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:50.685256 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:50.685580 (+ 324us) server_negotiation.cc:299] Negotiation successful
0618 23:31:50.685702 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 23:31:50.685966 392 external_mini_cluster.cc:1428] Setting key 89d3a6e763cd2e987e55344380f563ef
I20260618 23:31:50.686007 4162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:50.676370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32913 (local address 127.0.98.1:38875)
0618 23:31:50.676578 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:50.676620 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:50.677738 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:50.678013 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:50.678017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:50.678316 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:50.678528 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:50.678532 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:50.679255 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:50.679256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:50.680166 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:50.680169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:50.680577 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:50.680579 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:50.680710 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:50.681290 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:50.681298 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:50.682877 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:50.684433 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:50.684440 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:50.684442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:50.684703 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:50.684915 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:50.684915 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:50.684916 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:50.685027 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:50.685263 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:50.685268 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:50.685411 (+ 143us) client_negotiation.cc:769] Sending connection context
0618 23:31:50.685541 (+ 130us) client_negotiation.cc:240] Negotiation successful
0618 23:31:50.685742 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":78,"threads_started":1}
I20260618 23:31:50.686707 4160 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32913
I20260618 23:31:50.686836 4160 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:50.687055 4160 heartbeater.cc:511] Master 127.0.98.62:32913 requested a full tablet report, sending...
I20260618 23:31:50.687749 3969 ts_manager.cc:194] Registered new tserver with Master: f671c513452248fbb0b70a03aef0661f (127.0.98.1:42775)
I20260618 23:31:50.688642 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38875
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260618 23:31:50.714754 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32913
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:50.800731 4165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:50.800968 4165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:50.800994 4165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:50.802207 4165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:50.802254 4165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:50.802281 4165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:31:50.803802 4165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:50.804069 4165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:50.804327 4165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:50.809808 4173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:50.809855 4165 server_base.cc:1034] running on GCE node
W20260618 23:31:50.809859 4171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:50.809860 4170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:50.810427 4165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:50.810748 4165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:50.811935 4165 hybrid_clock.cc:648] HybridClock initialized: now 1781825510811925 us; error 33 us; skew 500 ppm
Jun 18 23:31:50 dist-test-slave-6224 krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:50.814941 4165 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:31:50.815591 4165 webserver.cc:460] Webserver started at http://127.0.98.2:44071/ using document root <none> and password file <none>
I20260618 23:31:50.815883 4165 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:50.815961 4165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:50.816150 4165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:50.817456 4165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "daae9339794f4114bf597f1c16c84c81"
format_stamp: "Formatted at 2026-06-18 23:31:50 on dist-test-slave-6224"
server_key: "640e6723a87f644e6a6ce75e557f78d1"
server_key_iv: "6904a21d26646daed7aca47c941ea32f"
server_key_version: "encryptionkey@0"
I20260618 23:31:50.817903 4165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "daae9339794f4114bf597f1c16c84c81"
format_stamp: "Formatted at 2026-06-18 23:31:50 on dist-test-slave-6224"
server_key: "640e6723a87f644e6a6ce75e557f78d1"
server_key_iv: "6904a21d26646daed7aca47c941ea32f"
server_key_version: "encryptionkey@0"
I20260618 23:31:50.819728 4165 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260618 23:31:50.820889 4180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:50.821134 4165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:50.821260 4165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "daae9339794f4114bf597f1c16c84c81"
format_stamp: "Formatted at 2026-06-18 23:31:50 on dist-test-slave-6224"
server_key: "640e6723a87f644e6a6ce75e557f78d1"
server_key_iv: "6904a21d26646daed7aca47c941ea32f"
server_key_version: "encryptionkey@0"
I20260618 23:31:50.821345 4165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:50.848562 4165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:50.851675 4165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:50.851888 4165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:50.852208 4165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:50.852618 4165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:50.852679 4165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:50.852728 4165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:50.852752 4165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:50.861194 4165 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40409
I20260618 23:31:50.861245 4293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40409 every 8 connection(s)
I20260618 23:31:50.861651 4165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
Jun 18 23:31:50 dist-test-slave-6224 krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825510, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:50.870932 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 4165
I20260618 23:31:50.871039 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:31:50.871322 392 external_mini_cluster.cc:1428] Setting key 4e244d0982554e644046cd747f5552fb
I20260618 23:31:50.873023 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:50.862574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60221 (local address 127.0.98.62:32913)
0618 23:31:50.862649 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:50.862651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:50.862892 (+ 241us) server_negotiation.cc:407] Connection header received
0618 23:31:50.863938 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:50.863940 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:50.863952 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:50.864053 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:50.864806 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:50.865350 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:50.866564 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:50.866783 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:50.866855 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:50.869290 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:50.869293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:50.869293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:50.869301 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:50.870684 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:50.871753 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:50.871754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:50.871755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:50.871835 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:50.872186 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:50.872187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:50.872187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:50.872348 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:50.872406 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:50.872866 (+ 460us) server_negotiation.cc:299] Negotiation successful
0618 23:31:50.872951 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 23:31:50.873409 4296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:50.862496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32913 (local address 127.0.98.2:60221)
0618 23:31:50.862705 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:50.862751 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:50.863793 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:50.864161 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:50.864164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:50.864470 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:50.864677 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:50.864681 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:50.865449 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:50.865450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:50.866447 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:50.866449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:50.866880 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:50.866881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:50.866981 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:50.867585 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:50.867596 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:50.869164 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:50.871195 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:50.871206 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:50.871209 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:50.871657 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:50.871924 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:50.871925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:50.871925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:50.872080 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:50.872422 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:50.872429 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:50.872647 (+ 218us) client_negotiation.cc:769] Sending connection context
0618 23:31:50.872808 (+ 161us) client_negotiation.cc:240] Negotiation successful
0618 23:31:50.873038 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":121,"threads_started":1}
I20260618 23:31:50.874279 4294 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32913
I20260618 23:31:50.874432 4294 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:50.874679 4294 heartbeater.cc:511] Master 127.0.98.62:32913 requested a full tablet report, sending...
I20260618 23:31:50.875165 3969 ts_manager.cc:194] Registered new tserver with Master: daae9339794f4114bf597f1c16c84c81 (127.0.98.2:40409)
I20260618 23:31:50.875701 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60221
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260618 23:31:50.899139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32913
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:50.990773 4299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:50.991068 4299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:50.991108 4299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:50.992527 4299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:50.992620 4299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:50.992671 4299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:31:50.994536 4299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:50.994858 4299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:50.995137 4299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:51.001125 4305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:51.001134 4304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:51.001134 4307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:51.001489 4299 server_base.cc:1034] running on GCE node
I20260618 23:31:51.001895 4299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:51.002220 4299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:51.003415 4299 hybrid_clock.cc:648] HybridClock initialized: now 1781825511003393 us; error 61 us; skew 500 ppm
Jun 18 23:31:51 dist-test-slave-6224 krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:51.006388 4299 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:31:51.006997 4299 webserver.cc:460] Webserver started at http://127.0.98.3:37047/ using document root <none> and password file <none>
I20260618 23:31:51.007282 4299 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:51.007359 4299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:51.007526 4299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:51.008785 4299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "6dc371baa01b4bfdb4ecd3458b6229d3"
format_stamp: "Formatted at 2026-06-18 23:31:51 on dist-test-slave-6224"
server_key: "606cd04e6513d3b65474779153dede58"
server_key_iv: "f181bdb2ba7da87d66b9b4887ac58cd8"
server_key_version: "encryptionkey@0"
I20260618 23:31:51.009189 4299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6dc371baa01b4bfdb4ecd3458b6229d3"
format_stamp: "Formatted at 2026-06-18 23:31:51 on dist-test-slave-6224"
server_key: "606cd04e6513d3b65474779153dede58"
server_key_iv: "f181bdb2ba7da87d66b9b4887ac58cd8"
server_key_version: "encryptionkey@0"
I20260618 23:31:51.010718 4299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:31:51.011698 4314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:51.012030 4299 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:51.012210 4299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "6dc371baa01b4bfdb4ecd3458b6229d3"
format_stamp: "Formatted at 2026-06-18 23:31:51 on dist-test-slave-6224"
server_key: "606cd04e6513d3b65474779153dede58"
server_key_iv: "f181bdb2ba7da87d66b9b4887ac58cd8"
server_key_version: "encryptionkey@0"
I20260618 23:31:51.012308 4299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:51.031116 4299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:51.033601 4299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:51.033773 4299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:51.034080 4299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:51.034452 4299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:51.034489 4299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:51.034514 4299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:51.034566 4299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:51.042471 4299 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45617
I20260618 23:31:51.042517 4427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45617 every 8 connection(s)
I20260618 23:31:51.042941 4299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:31:51.045090 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 4299
I20260618 23:31:51.045230 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:31:51.045473 392 external_mini_cluster.cc:1428] Setting key 4a46fa644f39f99c7e5e5dbb79f4f472
Jun 18 23:31:51 dist-test-slave-6224 krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825511, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:51.054260 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:51.043931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45565 (local address 127.0.98.62:32913)
0618 23:31:51.044028 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:51.044031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:51.044271 (+ 240us) server_negotiation.cc:407] Connection header received
0618 23:31:51.045677 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:51.045679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:51.045691 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:51.045719 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:51.046777 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:51.047406 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:51.048572 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:51.048757 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:51.048860 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:51.051184 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:51.051185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:51.051185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:51.051192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:51.052725 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:51.053206 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:51.053207 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:51.053207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:51.053260 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:51.053578 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:51.053579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:51.053579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:51.053697 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:51.053756 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:51.054106 (+ 350us) server_negotiation.cc:299] Negotiation successful
0618 23:31:51.054190 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 23:31:51.054514 4430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:51.043874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32913 (local address 127.0.98.3:45565)
0618 23:31:51.044081 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:51.044126 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:51.045463 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:51.045897 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:51.045902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:51.046365 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:51.046699 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:51.046704 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:51.047488 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:51.047489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:51.048467 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:51.048469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:51.048848 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:51.048849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:51.048962 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:51.049532 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:51.049541 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:51.051078 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:51.052821 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:51.052828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:51.052831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:51.053107 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:51.053354 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:51.053355 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:51.053355 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:51.053478 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:51.053756 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:51.053762 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:51.053915 (+ 153us) client_negotiation.cc:769] Sending connection context
0618 23:31:51.054026 (+ 111us) client_negotiation.cc:240] Negotiation successful
0618 23:31:51.054212 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":129,"threads_started":1}
I20260618 23:31:51.055169 4428 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32913
I20260618 23:31:51.055295 4428 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:51.055517 4428 heartbeater.cc:511] Master 127.0.98.62:32913 requested a full tablet report, sending...
I20260618 23:31:51.056102 3969 ts_manager.cc:194] Registered new tserver with Master: 6dc371baa01b4bfdb4ecd3458b6229d3 (127.0.98.3:45617)
I20260618 23:31:51.056543 3969 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:45565
I20260618 23:31:51.057646 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 23:31:51 dist-test-slave-6224 krb5kdc[3924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 23:31:51 dist-test-slave-6224 krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260618 23:31:51.083508 4435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:51.074868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39600 (local address 127.0.98.1:42775)
0618 23:31:51.075037 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:51.075044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:51.075062 (+ 18us) server_negotiation.cc:407] Connection header received
0618 23:31:51.075091 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:51.075093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:51.075109 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:51.075189 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:51.076032 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:51.076771 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:51.077583 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:51.077940 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:51.078008 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:51.080575 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:51.080582 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:51.080582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:51.080590 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:51.082297 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:51.082683 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:51.082686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:51.082686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:51.082773 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:51.082964 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:51.082964 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:51.082965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:51.083099 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:51.083207 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:51.083354 (+ 147us) server_negotiation.cc:299] Negotiation successful
0618 23:31:51.083425 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
Jun 18 23:31:51 dist-test-slave-6224 krb5kdc[3924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:51.092465 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:51.085268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49644 (local address 127.0.98.62:32913)
0618 23:31:51.085327 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:51.085330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:51.085343 (+ 13us) server_negotiation.cc:407] Connection header received
0618 23:31:51.085414 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:51.085415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:51.085425 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:51.085464 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:51.085919 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:51.086392 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:51.087081 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:51.087258 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:51.087317 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:51.089868 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:51.089869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:51.089869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:51.089875 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:51.091265 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:51.091717 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:51.091718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:51.091718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:51.091772 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:51.091964 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:51.091965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:51.091965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:51.092108 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:51.092188 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:51.092303 (+ 115us) server_negotiation.cc:299] Negotiation successful
0618 23:31:51.092368 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260618 23:31:51.092686 3969 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49644
I20260618 23:31:51.093151 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 4031
I20260618 23:31:51.099305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 4165
I20260618 23:31:51.104977 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 4299
I20260618 23:31:51.110354 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 3937
2026-06-18T23:31:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1944 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:31:51 dist-test-slave-6224 krb5kdc[4440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:31:51 dist-test-slave-6224 krb5kdc[4440](info): set up 2 sockets
Jun 18 23:31:51 dist-test-slave-6224 krb5kdc[4440](info): commencing operation
krb5kdc: starting...
W20260618 23:31:52.163673 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:31:52 dist-test-slave-6224 krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:31:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:31:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:31:52.292304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46883
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:52.385880 4453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:52.386133 4453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:52.386160 4453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:52.387578 4453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:31:52.387637 4453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:52.387652 4453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:52.387666 4453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:31:52.387679 4453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:31:52.389436 4453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:52.389741 4453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:52.390043 4453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:52.395773 4459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:52.395893 4461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:52.395773 4458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:52.395802 4453 server_base.cc:1034] running on GCE node
I20260618 23:31:52.396345 4453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:52.396714 4453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:52.397907 4453 hybrid_clock.cc:648] HybridClock initialized: now 1781825512397875 us; error 53 us; skew 500 ppm
Jun 18 23:31:52 dist-test-slave-6224 krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:52.400687 4453 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:31:52.401068 4453 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:52.401122 4453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:52.401221 4453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:52.402335 4453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "37e9808e524d46acb3fe305eae2c85dd"
format_stamp: "Formatted at 2026-06-18 23:31:52 on dist-test-slave-6224"
server_key: "c1f7c90c2c861808d226efbce354adaa"
server_key_iv: "9496bb42fee98c778ca24ac47ab38e47"
server_key_version: "encryptionkey@0"
I20260618 23:31:52.402657 4453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "37e9808e524d46acb3fe305eae2c85dd"
format_stamp: "Formatted at 2026-06-18 23:31:52 on dist-test-slave-6224"
server_key: "c1f7c90c2c861808d226efbce354adaa"
server_key_iv: "9496bb42fee98c778ca24ac47ab38e47"
server_key_version: "encryptionkey@0"
I20260618 23:31:52.404058 4453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260618 23:31:52.404896 4467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:52.405179 4453 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:52.405265 4453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "37e9808e524d46acb3fe305eae2c85dd"
format_stamp: "Formatted at 2026-06-18 23:31:52 on dist-test-slave-6224"
server_key: "c1f7c90c2c861808d226efbce354adaa"
server_key_iv: "9496bb42fee98c778ca24ac47ab38e47"
server_key_version: "encryptionkey@0"
I20260618 23:31:52.405333 4453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:52.428215 4453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:52.430379 4453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:52.430506 4453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:52.435168 4453 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46883
I20260618 23:31:52.435267 4519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46883 every 8 connection(s)
I20260618 23:31:52.435588 4453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:31:52.436766 4520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:52.439007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 4453
I20260618 23:31:52.439143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:31:52.439352 392 external_mini_cluster.cc:1428] Setting key ebdde32606ac3222f80cc596c97e8780
I20260618 23:31:52.439893 4520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd: Bootstrap starting.
I20260618 23:31:52.440634 4520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:52.440986 4520 log.cc:824] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:52.441996 4520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd: No bootstrap required, opened a new log
I20260618 23:31:52.444396 4520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9808e524d46acb3fe305eae2c85dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46883 } }
I20260618 23:31:52.444550 4520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:52.444573 4520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37e9808e524d46acb3fe305eae2c85dd, State: Initialized, Role: FOLLOWER
I20260618 23:31:52.444721 4520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9808e524d46acb3fe305eae2c85dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46883 } }
I20260618 23:31:52.444796 4520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:31:52.444829 4520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:31:52.444870 4520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:52.445711 4520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9808e524d46acb3fe305eae2c85dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46883 } }
I20260618 23:31:52.445837 4520 leader_election.cc:302] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37e9808e524d46acb3fe305eae2c85dd; no voters:
I20260618 23:31:52.446034 4520 leader_election.cc:288] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:31:52.446121 4525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:52.446367 4520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:31:52.446437 4525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd [term 1 LEADER]: Becoming Leader. State: Replica: 37e9808e524d46acb3fe305eae2c85dd, State: Running, Role: LEADER
I20260618 23:31:52.446704 4525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9808e524d46acb3fe305eae2c85dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46883 } }
I20260618 23:31:52.447402 4526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37e9808e524d46acb3fe305eae2c85dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9808e524d46acb3fe305eae2c85dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46883 } } }
I20260618 23:31:52.447513 4526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd [sys.catalog]: This master's current role is: LEADER
Jun 18 23:31:52 dist-test-slave-6224 krb5kdc[4440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825512, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:52.448132 4527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37e9808e524d46acb3fe305eae2c85dd. Latest consensus state: current_term: 1 leader_uuid: "37e9808e524d46acb3fe305eae2c85dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37e9808e524d46acb3fe305eae2c85dd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46883 } } }
W20260618 23:31:52.448638 4542 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260618 23:31:52.448866 4527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd [sys.catalog]: This master's current role is: LEADER
W20260618 23:31:52.448879 4542 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:31:52.449189 4534 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:31:52.449402 4534 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:31:52.451649 4534 catalog_manager.cc:1347] Generated new cluster ID: ab222039ced74fbfa991607415cd8037
I20260618 23:31:52.451707 4534 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:31:52.452661 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:52.440081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49698 (local address 127.0.98.62:46883)
0618 23:31:52.440290 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:52.440300 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:52.440334 (+ 34us) server_negotiation.cc:407] Connection header received
0618 23:31:52.441055 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:52.441067 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:52.441140 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:52.441340 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:52.442178 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:52.443124 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:52.444374 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:52.445066 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:52.445167 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:52.448455 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:52.448460 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:52.448461 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:52.448477 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:52.450728 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:52.451210 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:52.451214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:52.451215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:52.451316 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:52.451542 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:52.451543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:52.451543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:52.451899 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:52.452072 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:52.452274 (+ 202us) server_negotiation.cc:299] Negotiation successful
0618 23:31:52.452429 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":107,"threads_started":1}
I20260618 23:31:52.463790 4534 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:31:52.464483 4534 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:31:52.471848 4534 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd: Generated new TSK 0
I20260618 23:31:52.472149 4534 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260618 23:31:52.506476 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46883
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:52.598215 4547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:52.598440 4547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:52.598466 4547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:52.599704 4547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:52.599752 4547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:52.599781 4547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:31:52.601344 4547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:52.601614 4547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:52.601878 4547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:52.607594 4555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:52.607614 4552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:52.607646 4553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:52.607972 4547 server_base.cc:1034] running on GCE node
I20260618 23:31:52.608284 4547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:52.608563 4547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:52.609732 4547 hybrid_clock.cc:648] HybridClock initialized: now 1781825512609706 us; error 40 us; skew 500 ppm
Jun 18 23:31:52 dist-test-slave-6224 krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:52.612653 4547 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:31:52.613067 4547 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:52.613147 4547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:52.613306 4547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:52.614403 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "1afa9473686c4086ba910026618946fc"
format_stamp: "Formatted at 2026-06-18 23:31:52 on dist-test-slave-6224"
server_key: "8c0450efbb94b6ba82f733aa1090d240"
server_key_iv: "0f4b40739cfd6e6823dfcbc8b4ad7096"
server_key_version: "encryptionkey@0"
I20260618 23:31:52.614777 4547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1afa9473686c4086ba910026618946fc"
format_stamp: "Formatted at 2026-06-18 23:31:52 on dist-test-slave-6224"
server_key: "8c0450efbb94b6ba82f733aa1090d240"
server_key_iv: "0f4b40739cfd6e6823dfcbc8b4ad7096"
server_key_version: "encryptionkey@0"
I20260618 23:31:52.616343 4547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260618 23:31:52.617350 4561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:52.617682 4547 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:52.617813 4547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "1afa9473686c4086ba910026618946fc"
format_stamp: "Formatted at 2026-06-18 23:31:52 on dist-test-slave-6224"
server_key: "8c0450efbb94b6ba82f733aa1090d240"
server_key_iv: "0f4b40739cfd6e6823dfcbc8b4ad7096"
server_key_version: "encryptionkey@0"
I20260618 23:31:52.617936 4547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:52.638782 4547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:52.641113 4547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:52.641309 4547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:52.641564 4547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:52.641904 4547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:52.641969 4547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:52.642012 4547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:52.642055 4547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:52.649523 4547 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36651
I20260618 23:31:52.649637 4674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36651 every 8 connection(s)
I20260618 23:31:52.649926 4547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:31:52.652285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 4547
I20260618 23:31:52.652419 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:31:52.652642 392 external_mini_cluster.cc:1428] Setting key a62e7ac591be9c90a8dd19803abaf86a
Jun 18 23:31:52 dist-test-slave-6224 krb5kdc[4440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260618 23:31:52.661706 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:52.650782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34509 (local address 127.0.98.62:46883)
0618 23:31:52.650906 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:52.650909 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:52.651108 (+ 199us) server_negotiation.cc:407] Connection header received
0618 23:31:52.651830 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:52.651832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:52.651848 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:52.651908 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:52.653407 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:52.654021 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:52.655655 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:52.655911 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:52.656006 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:52.658591 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:52.658593 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:52.658593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:52.658601 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:52.660192 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:52.660639 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:52.660641 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:52.660642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:52.660710 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:52.660975 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:52.660976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:52.660977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:52.661113 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:52.661192 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:52.661488 (+ 296us) server_negotiation.cc:299] Negotiation successful
0618 23:31:52.661587 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260618 23:31:52.661914 4677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:52.650736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46883 (local address 127.0.98.1:34509)
0618 23:31:52.650939 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:52.650992 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:52.651704 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:52.651994 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:52.652001 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:52.652494 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:52.653269 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:52.653276 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:52.654144 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:52.654146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:52.655521 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:52.655524 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:52.656017 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:52.656019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:52.656164 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:52.656815 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:52.656825 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:52.658484 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:52.660299 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:52.660306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:52.660309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:52.660550 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:52.660787 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:52.660788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:52.660788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:52.660906 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:52.661189 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:52.661197 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:52.661347 (+ 150us) client_negotiation.cc:769] Sending connection context
0618 23:31:52.661468 (+ 121us) client_negotiation.cc:240] Negotiation successful
0618 23:31:52.661652 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
I20260618 23:31:52.662547 4675 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46883
I20260618 23:31:52.662691 4675 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:52.662935 4675 heartbeater.cc:511] Master 127.0.98.62:46883 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260618 23:31:52.663637 4484 ts_manager.cc:194] Registered new tserver with Master: 1afa9473686c4086ba910026618946fc (127.0.98.1:36651)
I20260618 23:31:52.664331 4484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260618 23:31:52.679232 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46883
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:52.760578 4680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:52.760816 4680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:52.760843 4680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:52.762040 4680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:52.762092 4680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:52.762120 4680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:31:52.763659 4680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:52.763919 4680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:52.764164 4680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:52.769490 4686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:52.769490 4688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:52.769526 4685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:52.769973 4680 server_base.cc:1034] running on GCE node
I20260618 23:31:52.770221 4680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:52.770490 4680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:52.771641 4680 hybrid_clock.cc:648] HybridClock initialized: now 1781825512771612 us; error 36 us; skew 500 ppm
Jun 18 23:31:52 dist-test-slave-6224 krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:52.774446 4680 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:31:52.774875 4680 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:52.774931 4680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:52.775094 4680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:52.776268 4680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "c781dff2b71d49b0997423a92593c065"
format_stamp: "Formatted at 2026-06-18 23:31:52 on dist-test-slave-6224"
server_key: "d728e39a551003cb5b4fb580eab888d1"
server_key_iv: "909b77cdccbd66cd270a57870eedc200"
server_key_version: "encryptionkey@0"
I20260618 23:31:52.776635 4680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c781dff2b71d49b0997423a92593c065"
format_stamp: "Formatted at 2026-06-18 23:31:52 on dist-test-slave-6224"
server_key: "d728e39a551003cb5b4fb580eab888d1"
server_key_iv: "909b77cdccbd66cd270a57870eedc200"
server_key_version: "encryptionkey@0"
I20260618 23:31:52.778015 4680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:31:52.778823 4694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:52.779073 4680 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:52.779182 4680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "c781dff2b71d49b0997423a92593c065"
format_stamp: "Formatted at 2026-06-18 23:31:52 on dist-test-slave-6224"
server_key: "d728e39a551003cb5b4fb580eab888d1"
server_key_iv: "909b77cdccbd66cd270a57870eedc200"
server_key_version: "encryptionkey@0"
I20260618 23:31:52.779263 4680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:52.807392 4680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:52.810032 4680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:52.810209 4680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:52.810467 4680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:52.810797 4680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:52.810861 4680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:52.810906 4680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:52.810935 4680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:52.817965 4680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36817
I20260618 23:31:52.818006 4807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36817 every 8 connection(s)
I20260618 23:31:52.818297 4680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
Jun 18 23:31:52 dist-test-slave-6224 krb5kdc[4440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:52.825667 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 4680
I20260618 23:31:52.825875 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:31:52.826118 392 external_mini_cluster.cc:1428] Setting key fd02c9b07f3a29e171659faac092a2fb
I20260618 23:31:52.829699 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:52.819218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55265 (local address 127.0.98.62:46883)
0618 23:31:52.819354 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:52.819358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:52.819566 (+ 208us) server_negotiation.cc:407] Connection header received
0618 23:31:52.820236 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:52.820238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:52.820252 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:52.820301 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:52.821505 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:52.822151 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:52.823237 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:52.823454 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:52.823562 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:52.826137 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:52.826138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:52.826139 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:52.826154 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:52.828143 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:52.828635 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:52.828637 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:52.828637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:52.828705 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:52.828987 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:52.828988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:52.828989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:52.829114 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:52.829190 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:52.829494 (+ 304us) server_negotiation.cc:299] Negotiation successful
0618 23:31:52.829590 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260618 23:31:52.829952 4810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:52.819130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46883 (local address 127.0.98.2:55265)
0618 23:31:52.819374 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:52.819432 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:52.820103 (+ 671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:52.820402 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:52.820406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:52.820737 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:52.821401 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:52.821407 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:52.822247 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:52.822247 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:52.823121 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:52.823122 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:52.823572 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:52.823573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:52.823706 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:52.824345 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:52.824354 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:52.826022 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:52.828252 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:52.828259 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:52.828262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:52.828550 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:52.828784 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:52.828785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:52.828785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:52.828909 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:52.829191 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:52.829199 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:52.829357 (+ 158us) client_negotiation.cc:769] Sending connection context
0618 23:31:52.829474 (+ 117us) client_negotiation.cc:240] Negotiation successful
0618 23:31:52.829653 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
I20260618 23:31:52.830451 4808 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46883
I20260618 23:31:52.830562 4808 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:52.830797 4808 heartbeater.cc:511] Master 127.0.98.62:46883 requested a full tablet report, sending...
I20260618 23:31:52.831264 4484 ts_manager.cc:194] Registered new tserver with Master: c781dff2b71d49b0997423a92593c065 (127.0.98.2:36817)
I20260618 23:31:52.831774 4484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55265
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260618 23:31:52.854653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46883
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:52.936223 4813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:52.936460 4813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:52.936487 4813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:52.937736 4813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:52.937791 4813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:52.937820 4813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:31:52.939424 4813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:52.939689 4813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:52.939922 4813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:52.946067 4821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:52.946061 4818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:52.946053 4819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:52.946789 4813 server_base.cc:1034] running on GCE node
I20260618 23:31:52.947042 4813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:52.947345 4813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:52.948557 4813 hybrid_clock.cc:648] HybridClock initialized: now 1781825512948540 us; error 37 us; skew 500 ppm
Jun 18 23:31:52 dist-test-slave-6224 krb5kdc[4440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:52.951836 4813 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:31:52.952322 4813 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:52.952404 4813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:52.952569 4813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:52.953754 4813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "b9e3ecf9b03046e7b41ae2b6dad86dd2"
format_stamp: "Formatted at 2026-06-18 23:31:52 on dist-test-slave-6224"
server_key: "38a5e426473c5401ce261a260f0bf907"
server_key_iv: "6f6188bf583d18110f1a030ccc346e89"
server_key_version: "encryptionkey@0"
I20260618 23:31:52.954149 4813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b9e3ecf9b03046e7b41ae2b6dad86dd2"
format_stamp: "Formatted at 2026-06-18 23:31:52 on dist-test-slave-6224"
server_key: "38a5e426473c5401ce261a260f0bf907"
server_key_iv: "6f6188bf583d18110f1a030ccc346e89"
server_key_version: "encryptionkey@0"
I20260618 23:31:52.955744 4813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:31:52.956753 4827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:52.957091 4813 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:52.957233 4813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "b9e3ecf9b03046e7b41ae2b6dad86dd2"
format_stamp: "Formatted at 2026-06-18 23:31:52 on dist-test-slave-6224"
server_key: "38a5e426473c5401ce261a260f0bf907"
server_key_iv: "6f6188bf583d18110f1a030ccc346e89"
server_key_version: "encryptionkey@0"
I20260618 23:31:52.957329 4813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:52.972415 4813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:52.974538 4813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:52.974712 4813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:52.974996 4813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:52.975334 4813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:52.975392 4813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:52.975435 4813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:52.975462 4813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:52.982556 4813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37879
I20260618 23:31:52.982604 4940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37879 every 8 connection(s)
I20260618 23:31:52.982883 4813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
Jun 18 23:31:52 dist-test-slave-6224 krb5kdc[4440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825512, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:52.991149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 4813
I20260618 23:31:52.991271 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:31:52.991438 392 external_mini_cluster.cc:1428] Setting key 128fce0c6d167e2be40c300c2521d32d
I20260618 23:31:52.993736 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:52.983807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59091 (local address 127.0.98.62:46883)
0618 23:31:52.983864 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:52.983866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:52.984128 (+ 262us) server_negotiation.cc:407] Connection header received
0618 23:31:52.984788 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:52.984789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:52.984803 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:52.984861 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:52.985961 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:52.986515 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:52.987662 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:52.987890 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:52.987979 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:52.990337 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:52.990338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:52.990339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:52.990347 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:52.992004 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:52.992634 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:52.992636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:52.992636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:52.992716 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:52.992993 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:52.992994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:52.992995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:52.993133 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:52.993208 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:52.993530 (+ 322us) server_negotiation.cc:299] Negotiation successful
0618 23:31:52.993634 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260618 23:31:52.993960 4943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:52.983775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46883 (local address 127.0.98.3:59091)
0618 23:31:52.983953 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:52.984012 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:52.984658 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:52.984963 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:52.984968 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:52.985292 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:52.985859 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:52.985863 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:52.986614 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:52.986615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:52.987491 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:52.987494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:52.987989 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:52.987990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:52.988130 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:52.988694 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:52.988704 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:52.990234 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:52.992107 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:52.992120 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:52.992123 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:52.992552 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:52.992793 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:52.992794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:52.992794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:52.992912 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:52.993206 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:52.993213 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:52.993380 (+ 167us) client_negotiation.cc:769] Sending connection context
0618 23:31:52.993509 (+ 129us) client_negotiation.cc:240] Negotiation successful
0618 23:31:52.993707 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20260618 23:31:52.994534 4941 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46883
I20260618 23:31:52.994645 4941 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:52.994805 4941 heartbeater.cc:511] Master 127.0.98.62:46883 requested a full tablet report, sending...
I20260618 23:31:52.995311 4484 ts_manager.cc:194] Registered new tserver with Master: b9e3ecf9b03046e7b41ae2b6dad86dd2 (127.0.98.3:37879)
I20260618 23:31:52.995776 4484 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59091
I20260618 23:31:53.004020 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:31:53.012313 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.005492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49710 (local address 127.0.98.62:46883)
0618 23:31:53.005567 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:53.005570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:53.005703 (+ 133us) server_negotiation.cc:407] Connection header received
0618 23:31:53.005787 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:53.005788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:53.005806 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:53.005912 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:53.006502 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.007004 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.007875 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.008064 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.008169 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:53.009437 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:53.009439 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:53.009439 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:53.009446 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:53.010898 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:53.011320 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:53.011321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:53.011322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:53.011385 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:53.011630 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:53.011631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:53.011631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:53.011797 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:53.011961 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:53.012108 (+ 147us) server_negotiation.cc:299] Negotiation successful
0618 23:31:53.012201 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260618 23:31:53.014464 4484 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:31:53.016638 4484 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:31:53.024824 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.019784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36817 (local address 127.0.0.1:45830)
0618 23:31:53.020280 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:53.020294 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:53.020355 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:53.020666 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:53.020668 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:53.020671 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:53.020743 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:53.020746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.021885 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.021886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:53.023384 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:53.023387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.024460 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.024461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:53.024568 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:53.024585 (+ 17us) client_negotiation.cc:769] Sending connection context
0618 23:31:53.024656 (+ 71us) client_negotiation.cc:240] Negotiation successful
0618 23:31:53.024730 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":33,"spinlock_wait_cycles":768,"thread_start_us":128,"threads_started":1}
I20260618 23:31:53.024835 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.020095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51440 (local address 127.0.98.1:36651)
0618 23:31:53.020318 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:53.020323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:53.020545 (+ 222us) server_negotiation.cc:407] Connection header received
0618 23:31:53.020596 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:53.020598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:53.020613 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:53.020679 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:53.020925 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.022034 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.023495 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.024280 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.024354 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:53.024418 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:53.024705 (+ 287us) server_negotiation.cc:299] Negotiation successful
0618 23:31:53.024744 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":88,"threads_started":1}
I20260618 23:31:53.025123 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.019784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36651 (local address 127.0.0.1:51440)
0618 23:31:53.020420 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:53.020429 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:53.020478 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:53.020769 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:53.020770 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:53.020772 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:53.020815 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:53.020817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.022160 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.022161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:53.023384 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:53.023387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.024459 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.024460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:53.024568 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:53.024585 (+ 17us) client_negotiation.cc:769] Sending connection context
0618 23:31:53.024656 (+ 71us) client_negotiation.cc:240] Negotiation successful
0618 23:31:53.025050 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":373,"thread_start_us":149,"threads_started":1}
I20260618 23:31:53.025336 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.020095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45830 (local address 127.0.98.2:36817)
0618 23:31:53.020355 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:53.020361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:53.020373 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:31:53.020468 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:53.020470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:53.020489 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:53.020569 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:53.020824 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.021754 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.023489 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.024280 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.024354 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:53.024418 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:53.025219 (+ 801us) server_negotiation.cc:299] Negotiation successful
0618 23:31:53.025262 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":88,"threads_started":1}
I20260618 23:31:53.026023 4742 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb7061c69f5477485dbb2fc893a3d40 (DEFAULT_TABLE table=test-table [id=5a5edf82232f4c698f1987ff155a7e08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:53.026396 4742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb7061c69f5477485dbb2fc893a3d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:53.026511 4609 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb7061c69f5477485dbb2fc893a3d40 (DEFAULT_TABLE table=test-table [id=5a5edf82232f4c698f1987ff155a7e08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:53.026954 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb7061c69f5477485dbb2fc893a3d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:53.027076 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.019781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37879 (local address 127.0.0.1:45906)
0618 23:31:53.019983 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:53.020033 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:53.020112 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:53.020569 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:53.020572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:53.020576 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:53.020698 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:53.020702 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.023089 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.023090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:53.025180 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:53.025183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.026811 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.026813 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:53.026892 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:53.026899 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 23:31:53.026933 (+ 34us) client_negotiation.cc:240] Negotiation successful
0618 23:31:53.026995 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":63,"threads_started":1}
I20260618 23:31:53.027509 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.019897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45906 (local address 127.0.98.3:37879)
0618 23:31:53.020295 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:53.020301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:53.020313 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:31:53.020360 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:53.020363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:53.020380 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:53.020806 (+ 426us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:53.020922 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.022961 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.025936 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.026687 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.027260 (+ 573us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:53.027334 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:53.027361 (+ 27us) server_negotiation.cc:299] Negotiation successful
0618 23:31:53.027409 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":120,"threads_started":1}
I20260618 23:31:53.028286 4875 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb7061c69f5477485dbb2fc893a3d40 (DEFAULT_TABLE table=test-table [id=5a5edf82232f4c698f1987ff155a7e08]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:53.028661 4875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb7061c69f5477485dbb2fc893a3d40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:53.030009 4958 tablet_bootstrap.cc:492] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065: Bootstrap starting.
I20260618 23:31:53.030421 4959 tablet_bootstrap.cc:492] T 8eb7061c69f5477485dbb2fc893a3d40 P 1afa9473686c4086ba910026618946fc: Bootstrap starting.
I20260618 23:31:53.030901 4958 tablet_bootstrap.cc:654] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:53.031134 4959 tablet_bootstrap.cc:654] T 8eb7061c69f5477485dbb2fc893a3d40 P 1afa9473686c4086ba910026618946fc: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:53.031455 4958 log.cc:824] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:53.031780 4959 log.cc:824] T 8eb7061c69f5477485dbb2fc893a3d40 P 1afa9473686c4086ba910026618946fc: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:53.032359 4958 tablet_bootstrap.cc:492] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065: No bootstrap required, opened a new log
I20260618 23:31:53.032407 4962 tablet_bootstrap.cc:492] T 8eb7061c69f5477485dbb2fc893a3d40 P b9e3ecf9b03046e7b41ae2b6dad86dd2: Bootstrap starting.
I20260618 23:31:53.032449 4958 ts_tablet_manager.cc:1397] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 23:31:53.033102 4962 tablet_bootstrap.cc:654] T 8eb7061c69f5477485dbb2fc893a3d40 P b9e3ecf9b03046e7b41ae2b6dad86dd2: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:53.033713 4962 log.cc:824] T 8eb7061c69f5477485dbb2fc893a3d40 P b9e3ecf9b03046e7b41ae2b6dad86dd2: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:53.034518 4959 tablet_bootstrap.cc:492] T 8eb7061c69f5477485dbb2fc893a3d40 P 1afa9473686c4086ba910026618946fc: No bootstrap required, opened a new log
I20260618 23:31:53.034531 4962 tablet_bootstrap.cc:492] T 8eb7061c69f5477485dbb2fc893a3d40 P b9e3ecf9b03046e7b41ae2b6dad86dd2: No bootstrap required, opened a new log
I20260618 23:31:53.034601 4962 ts_tablet_manager.cc:1397] T 8eb7061c69f5477485dbb2fc893a3d40 P b9e3ecf9b03046e7b41ae2b6dad86dd2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:31:53.034664 4959 ts_tablet_manager.cc:1397] T 8eb7061c69f5477485dbb2fc893a3d40 P 1afa9473686c4086ba910026618946fc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260618 23:31:53.035079 4958 raft_consensus.cc:348] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e3ecf9b03046e7b41ae2b6dad86dd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37879 } } peers { permanent_uuid: "c781dff2b71d49b0997423a92593c065" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36817 } } peers { permanent_uuid: "1afa9473686c4086ba910026618946fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36651 } }
I20260618 23:31:53.035233 4958 raft_consensus.cc:374] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:53.035256 4958 raft_consensus.cc:729] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c781dff2b71d49b0997423a92593c065, State: Initialized, Role: FOLLOWER
I20260618 23:31:53.035460 4958 consensus_queue.cc:260] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e3ecf9b03046e7b41ae2b6dad86dd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37879 } } peers { permanent_uuid: "c781dff2b71d49b0997423a92593c065" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36817 } } peers { permanent_uuid: "1afa9473686c4086ba910026618946fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36651 } }
I20260618 23:31:53.035813 4958 ts_tablet_manager.cc:1428] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:31:53.036001 4808 heartbeater.cc:503] Master 127.0.98.62:46883 was elected leader, sending a full tablet report...
I20260618 23:31:53.036690 4962 raft_consensus.cc:348] T 8eb7061c69f5477485dbb2fc893a3d40 P b9e3ecf9b03046e7b41ae2b6dad86dd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e3ecf9b03046e7b41ae2b6dad86dd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37879 } } peers { permanent_uuid: "c781dff2b71d49b0997423a92593c065" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36817 } } peers { permanent_uuid: "1afa9473686c4086ba910026618946fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36651 } }
I20260618 23:31:53.036911 4962 raft_consensus.cc:374] T 8eb7061c69f5477485dbb2fc893a3d40 P b9e3ecf9b03046e7b41ae2b6dad86dd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:53.036948 4962 raft_consensus.cc:729] T 8eb7061c69f5477485dbb2fc893a3d40 P b9e3ecf9b03046e7b41ae2b6dad86dd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9e3ecf9b03046e7b41ae2b6dad86dd2, State: Initialized, Role: FOLLOWER
I20260618 23:31:53.037129 4962 consensus_queue.cc:260] T 8eb7061c69f5477485dbb2fc893a3d40 P b9e3ecf9b03046e7b41ae2b6dad86dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e3ecf9b03046e7b41ae2b6dad86dd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37879 } } peers { permanent_uuid: "c781dff2b71d49b0997423a92593c065" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36817 } } peers { permanent_uuid: "1afa9473686c4086ba910026618946fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36651 } }
I20260618 23:31:53.037097 4959 raft_consensus.cc:348] T 8eb7061c69f5477485dbb2fc893a3d40 P 1afa9473686c4086ba910026618946fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e3ecf9b03046e7b41ae2b6dad86dd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37879 } } peers { permanent_uuid: "c781dff2b71d49b0997423a92593c065" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36817 } } peers { permanent_uuid: "1afa9473686c4086ba910026618946fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36651 } }
I20260618 23:31:53.037231 4959 raft_consensus.cc:374] T 8eb7061c69f5477485dbb2fc893a3d40 P 1afa9473686c4086ba910026618946fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:53.037267 4959 raft_consensus.cc:729] T 8eb7061c69f5477485dbb2fc893a3d40 P 1afa9473686c4086ba910026618946fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1afa9473686c4086ba910026618946fc, State: Initialized, Role: FOLLOWER
I20260618 23:31:53.037415 4959 consensus_queue.cc:260] T 8eb7061c69f5477485dbb2fc893a3d40 P 1afa9473686c4086ba910026618946fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e3ecf9b03046e7b41ae2b6dad86dd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37879 } } peers { permanent_uuid: "c781dff2b71d49b0997423a92593c065" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36817 } } peers { permanent_uuid: "1afa9473686c4086ba910026618946fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36651 } }
I20260618 23:31:53.037475 4941 heartbeater.cc:503] Master 127.0.98.62:46883 was elected leader, sending a full tablet report...
I20260618 23:31:53.037473 4962 ts_tablet_manager.cc:1428] T 8eb7061c69f5477485dbb2fc893a3d40 P b9e3ecf9b03046e7b41ae2b6dad86dd2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:31:53.037729 4959 ts_tablet_manager.cc:1428] T 8eb7061c69f5477485dbb2fc893a3d40 P 1afa9473686c4086ba910026618946fc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:31:53.037904 4675 heartbeater.cc:503] Master 127.0.98.62:46883 was elected leader, sending a full tablet report...
W20260618 23:31:53.068929 4809 tablet.cc:2369] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:31:53.150751 4676 tablet.cc:2369] T 8eb7061c69f5477485dbb2fc893a3d40 P 1afa9473686c4086ba910026618946fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:31:53.233532 4942 tablet.cc:2369] T 8eb7061c69f5477485dbb2fc893a3d40 P b9e3ecf9b03046e7b41ae2b6dad86dd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:31:53.294641 4964 raft_consensus.cc:482] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:31:53.294832 4964 raft_consensus.cc:504] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e3ecf9b03046e7b41ae2b6dad86dd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37879 } } peers { permanent_uuid: "c781dff2b71d49b0997423a92593c065" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36817 } } peers { permanent_uuid: "1afa9473686c4086ba910026618946fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36651 } }
I20260618 23:31:53.295290 4964 leader_election.cc:288] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9e3ecf9b03046e7b41ae2b6dad86dd2 (127.0.98.3:37879), 1afa9473686c4086ba910026618946fc (127.0.98.1:36651)
I20260618 23:31:53.299852 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.295669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54275 (local address 127.0.98.3:37879)
0618 23:31:53.295762 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:53.295765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:53.295779 (+ 14us) server_negotiation.cc:407] Connection header received
0618 23:31:53.295813 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:53.295814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:53.295828 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:53.295890 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:53.296146 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.296936 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.298739 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.299374 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.299454 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:53.299466 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:53.299726 (+ 260us) server_negotiation.cc:299] Negotiation successful
0618 23:31:53.299771 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260618 23:31:53.299899 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.295678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52635 (local address 127.0.98.1:36651)
0618 23:31:53.295738 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:53.295740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:53.295863 (+ 123us) server_negotiation.cc:407] Connection header received
0618 23:31:53.295974 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:53.295976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:53.295991 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:53.296049 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:31:53.296249 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.297012 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.298738 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.299361 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.299454 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:53.299466 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:53.299726 (+ 260us) server_negotiation.cc:299] Negotiation successful
0618 23:31:53.299770 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260618 23:31:53.299887 4810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.295528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37879 (local address 127.0.98.2:54275)
0618 23:31:53.295648 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:53.295663 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:53.295718 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:53.295927 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:53.295929 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:53.295932 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:53.296048 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:53.296051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.297040 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.297041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:53.298578 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:53.298580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.299477 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.299478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:53.299618 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:53.299621 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:31:53.299678 (+ 57us) client_negotiation.cc:240] Negotiation successful
0618 23:31:53.299734 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":60}
I20260618 23:31:53.299894 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.295600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36651 (local address 127.0.98.2:52635)
0618 23:31:53.295748 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:53.295762 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:53.295858 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:53.296072 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:53.296074 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:53.296076 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:31:53.296157 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:53.296159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.297099 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.297099 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:53.298572 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:53.298576 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.299477 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.299478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:53.299602 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:53.299607 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:31:53.299678 (+ 71us) client_negotiation.cc:240] Negotiation successful
0618 23:31:53.299731 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":10,"thread_start_us":69,"threads_started":1}
I20260618 23:31:53.300465 4895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb7061c69f5477485dbb2fc893a3d40" candidate_uuid: "c781dff2b71d49b0997423a92593c065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9e3ecf9b03046e7b41ae2b6dad86dd2" is_pre_election: true
I20260618 23:31:53.300478 4629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb7061c69f5477485dbb2fc893a3d40" candidate_uuid: "c781dff2b71d49b0997423a92593c065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1afa9473686c4086ba910026618946fc" is_pre_election: true
I20260618 23:31:53.300683 4629 raft_consensus.cc:2454] T 8eb7061c69f5477485dbb2fc893a3d40 P 1afa9473686c4086ba910026618946fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c781dff2b71d49b0997423a92593c065 in term 0.
I20260618 23:31:53.300683 4895 raft_consensus.cc:2454] T 8eb7061c69f5477485dbb2fc893a3d40 P b9e3ecf9b03046e7b41ae2b6dad86dd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c781dff2b71d49b0997423a92593c065 in term 0.
I20260618 23:31:53.300982 4695 leader_election.cc:302] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1afa9473686c4086ba910026618946fc, c781dff2b71d49b0997423a92593c065; no voters:
I20260618 23:31:53.301216 4964 raft_consensus.cc:2781] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:31:53.301285 4964 raft_consensus.cc:482] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:31:53.301311 4964 raft_consensus.cc:3037] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:53.302112 4964 raft_consensus.cc:504] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e3ecf9b03046e7b41ae2b6dad86dd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37879 } } peers { permanent_uuid: "c781dff2b71d49b0997423a92593c065" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36817 } } peers { permanent_uuid: "1afa9473686c4086ba910026618946fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36651 } }
I20260618 23:31:53.302285 4964 leader_election.cc:288] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065 [CANDIDATE]: Term 1 election: Requested vote from peers b9e3ecf9b03046e7b41ae2b6dad86dd2 (127.0.98.3:37879), 1afa9473686c4086ba910026618946fc (127.0.98.1:36651)
I20260618 23:31:53.302520 4629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb7061c69f5477485dbb2fc893a3d40" candidate_uuid: "c781dff2b71d49b0997423a92593c065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1afa9473686c4086ba910026618946fc"
I20260618 23:31:53.302520 4895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eb7061c69f5477485dbb2fc893a3d40" candidate_uuid: "c781dff2b71d49b0997423a92593c065" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9e3ecf9b03046e7b41ae2b6dad86dd2"
I20260618 23:31:53.302615 4629 raft_consensus.cc:3037] T 8eb7061c69f5477485dbb2fc893a3d40 P 1afa9473686c4086ba910026618946fc [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:53.302614 4895 raft_consensus.cc:3037] T 8eb7061c69f5477485dbb2fc893a3d40 P b9e3ecf9b03046e7b41ae2b6dad86dd2 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:53.303640 4629 raft_consensus.cc:2454] T 8eb7061c69f5477485dbb2fc893a3d40 P 1afa9473686c4086ba910026618946fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c781dff2b71d49b0997423a92593c065 in term 1.
I20260618 23:31:53.303640 4895 raft_consensus.cc:2454] T 8eb7061c69f5477485dbb2fc893a3d40 P b9e3ecf9b03046e7b41ae2b6dad86dd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c781dff2b71d49b0997423a92593c065 in term 1.
I20260618 23:31:53.303908 4695 leader_election.cc:302] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1afa9473686c4086ba910026618946fc, c781dff2b71d49b0997423a92593c065; no voters:
I20260618 23:31:53.304111 4964 raft_consensus.cc:2781] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:53.304337 4964 raft_consensus.cc:686] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065 [term 1 LEADER]: Becoming Leader. State: Replica: c781dff2b71d49b0997423a92593c065, State: Running, Role: LEADER
I20260618 23:31:53.304471 4964 consensus_queue.cc:237] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e3ecf9b03046e7b41ae2b6dad86dd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37879 } } peers { permanent_uuid: "c781dff2b71d49b0997423a92593c065" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36817 } } peers { permanent_uuid: "1afa9473686c4086ba910026618946fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36651 } }
I20260618 23:31:53.305615 4484 catalog_manager.cc:5495] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065 reported cstate change: term changed from 0 to 1, leader changed from <none> to c781dff2b71d49b0997423a92593c065 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c781dff2b71d49b0997423a92593c065" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9e3ecf9b03046e7b41ae2b6dad86dd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c781dff2b71d49b0997423a92593c065" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1afa9473686c4086ba910026618946fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36651 } health_report { overall_health: UNKNOWN } } }
I20260618 23:31:53.313182 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.310373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45840 (local address 127.0.98.2:36817)
0618 23:31:53.310446 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:53.310448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:53.310459 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:31:53.310518 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:53.310518 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:53.310536 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:53.310597 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:31:53.310795 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.311314 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.312256 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:53.312448 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:53.312520 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:53.312739 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:31:53.312882 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:31:53.312956 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:53.313039 (+ 83us) server_negotiation.cc:299] Negotiation successful
0618 23:31:53.313059 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 23:31:53.315421 4895 raft_consensus.cc:1261] T 8eb7061c69f5477485dbb2fc893a3d40 P b9e3ecf9b03046e7b41ae2b6dad86dd2 [term 1 FOLLOWER]: Refusing update from remote peer c781dff2b71d49b0997423a92593c065: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:31:53.315454 4629 raft_consensus.cc:1261] T 8eb7061c69f5477485dbb2fc893a3d40 P 1afa9473686c4086ba910026618946fc [term 1 FOLLOWER]: Refusing update from remote peer c781dff2b71d49b0997423a92593c065: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:31:53.315927 4964 consensus_queue.cc:1035] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1afa9473686c4086ba910026618946fc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:31:53.316098 4968 consensus_queue.cc:1035] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9e3ecf9b03046e7b41ae2b6dad86dd2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:31:53.318863 4971 mvcc.cc:205] Tried to move back new op lower bound from 7298357302536577024 to 7298357302496202752. Current Snapshot: MvccSnapshot[applied={T|T < 7298357302536577024}]
I20260618 23:31:53.319682 4973 mvcc.cc:205] Tried to move back new op lower bound from 7298357302536577024 to 7298357302496202752. Current Snapshot: MvccSnapshot[applied={T|T < 7298357302536577024}]
I20260618 23:31:53.319300 4972 mvcc.cc:205] Tried to move back new op lower bound from 7298357302536577024 to 7298357302496202752. Current Snapshot: MvccSnapshot[applied={T|T < 7298357302536577024}]
I20260618 23:31:53.321722 4482 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:31:53.321930 4482 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:31:53.322957 4482 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37e9808e524d46acb3fe305eae2c85dd: Sending DeleteTablet for 3 replicas of tablet 8eb7061c69f5477485dbb2fc893a3d40
I20260618 23:31:53.323426 4875 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb7061c69f5477485dbb2fc893a3d40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45906
I20260618 23:31:53.323441 4742 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb7061c69f5477485dbb2fc893a3d40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45830
I20260618 23:31:53.323709 4609 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eb7061c69f5477485dbb2fc893a3d40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51440
I20260618 23:31:53.323899 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 4547
I20260618 23:31:53.324882 4981 tablet_replica.cc:330] T 8eb7061c69f5477485dbb2fc893a3d40 P b9e3ecf9b03046e7b41ae2b6dad86dd2: stopping tablet replica
I20260618 23:31:53.325057 4981 raft_consensus.cc:2229] T 8eb7061c69f5477485dbb2fc893a3d40 P b9e3ecf9b03046e7b41ae2b6dad86dd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:31:53.325268 4981 raft_consensus.cc:2258] T 8eb7061c69f5477485dbb2fc893a3d40 P b9e3ecf9b03046e7b41ae2b6dad86dd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:53.327117 4981 ts_tablet_manager.cc:1905] T 8eb7061c69f5477485dbb2fc893a3d40 P b9e3ecf9b03046e7b41ae2b6dad86dd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:31:53.329054 4981 ts_tablet_manager.cc:1918] T 8eb7061c69f5477485dbb2fc893a3d40 P b9e3ecf9b03046e7b41ae2b6dad86dd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:31:53.329191 4981 log.cc:1196] T 8eb7061c69f5477485dbb2fc893a3d40 P b9e3ecf9b03046e7b41ae2b6dad86dd2: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/8eb7061c69f5477485dbb2fc893a3d40
I20260618 23:31:53.329485 4981 ts_tablet_manager.cc:1939] T 8eb7061c69f5477485dbb2fc893a3d40 P b9e3ecf9b03046e7b41ae2b6dad86dd2: Deleting consensus metadata
I20260618 23:31:53.330070 4469 catalog_manager.cc:4841] TS b9e3ecf9b03046e7b41ae2b6dad86dd2 (127.0.98.3:37879): tablet 8eb7061c69f5477485dbb2fc893a3d40 (table test-table [id=5a5edf82232f4c698f1987ff155a7e08]) successfully deleted
I20260618 23:31:53.331797 4982 tablet_replica.cc:330] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065: stopping tablet replica
I20260618 23:31:53.331975 4982 raft_consensus.cc:2229] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:31:53.332180 4982 raft_consensus.cc:2258] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:53.332731 4982 ts_tablet_manager.cc:1905] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:31:53.334347 4982 ts_tablet_manager.cc:1918] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:31:53.334445 4982 log.cc:1196] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/8eb7061c69f5477485dbb2fc893a3d40
I20260618 23:31:53.334718 4982 ts_tablet_manager.cc:1939] T 8eb7061c69f5477485dbb2fc893a3d40 P c781dff2b71d49b0997423a92593c065: Deleting consensus metadata
I20260618 23:31:53.335289 4470 catalog_manager.cc:4841] TS c781dff2b71d49b0997423a92593c065 (127.0.98.2:36817): tablet 8eb7061c69f5477485dbb2fc893a3d40 (table test-table [id=5a5edf82232f4c698f1987ff155a7e08]) successfully deleted
W20260618 23:31:53.335590 4468 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36651 (error 108)
I20260618 23:31:53.335875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 4680
I20260618 23:31:53.336779 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.336634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36651 (local address 127.0.0.1:51454)
0618 23:31:53.336693 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:53.336735 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260618 23:31:53.336930 4468 catalog_manager.cc:4584] TS 1afa9473686c4086ba910026618946fc (127.0.98.1:36651): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8eb7061c69f5477485dbb2fc893a3d40: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36651: connect: Connection refused (error 111)
I20260618 23:31:53.341409 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 4813
I20260618 23:31:53.347338 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 4453
2026-06-18T23:31:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2236 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-18T23:31:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:31:53Z Disabled control of system clock
I20260618 23:31:53.377447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38289
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260618 23:31:53.461719 4988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:53.461985 4988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:53.462013 4988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:53.463222 4988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:31:53.463274 4988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:53.463287 4988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:53.463301 4988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:31:53.463313 4988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:31:53.464775 4988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38289
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:53.465032 4988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:53.465332 4988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:53.470806 4994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:53.470973 4993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:53.471037 4988 server_base.cc:1034] running on GCE node
W20260618 23:31:53.470866 4996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:53.471478 4988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:53.471865 4988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:53.473067 4988 hybrid_clock.cc:648] HybridClock initialized: now 1781825513473044 us; error 44 us; skew 500 ppm
I20260618 23:31:53.474510 4988 webserver.cc:460] Webserver started at http://127.0.98.62:42423/ using document root <none> and password file <none>
I20260618 23:31:53.474789 4988 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:53.474874 4988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:53.475045 4988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:53.476333 4988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "a71c6b1ca3414a1ab943e628d60a5846"
format_stamp: "Formatted at 2026-06-18 23:31:53 on dist-test-slave-6224"
server_key: "c97389b5ff45ba233b09d98077aa35ff"
server_key_iv: "04f29ba08ab94f6da2ac10e183a56446"
server_key_version: "encryptionkey@0"
I20260618 23:31:53.476725 4988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "a71c6b1ca3414a1ab943e628d60a5846"
format_stamp: "Formatted at 2026-06-18 23:31:53 on dist-test-slave-6224"
server_key: "c97389b5ff45ba233b09d98077aa35ff"
server_key_iv: "04f29ba08ab94f6da2ac10e183a56446"
server_key_version: "encryptionkey@0"
I20260618 23:31:53.478222 4988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:31:53.479110 5002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:53.479426 4988 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:53.479554 4988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "a71c6b1ca3414a1ab943e628d60a5846"
format_stamp: "Formatted at 2026-06-18 23:31:53 on dist-test-slave-6224"
server_key: "c97389b5ff45ba233b09d98077aa35ff"
server_key_iv: "04f29ba08ab94f6da2ac10e183a56446"
server_key_version: "encryptionkey@0"
I20260618 23:31:53.479681 4988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:53.483238 4988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:53.483601 4988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:53.483742 4988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:53.488243 4988 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38289
I20260618 23:31:53.488303 5054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38289 every 8 connection(s)
I20260618 23:31:53.488680 4988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:31:53.489495 5055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:53.492472 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 4988
I20260618 23:31:53.492606 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:31:53.492807 392 external_mini_cluster.cc:1428] Setting key e359a39fd56f90091123f3aa5d801fd5
I20260618 23:31:53.493111 5055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846: Bootstrap starting.
I20260618 23:31:53.493957 5055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:53.494305 5055 log.cc:824] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:53.495114 5055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846: No bootstrap required, opened a new log
I20260618 23:31:53.496930 5055 raft_consensus.cc:348] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71c6b1ca3414a1ab943e628d60a5846" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38289 } }
I20260618 23:31:53.497105 5055 raft_consensus.cc:374] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:53.497130 5055 raft_consensus.cc:729] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a71c6b1ca3414a1ab943e628d60a5846, State: Initialized, Role: FOLLOWER
I20260618 23:31:53.497295 5055 consensus_queue.cc:260] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71c6b1ca3414a1ab943e628d60a5846" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38289 } }
I20260618 23:31:53.497372 5055 raft_consensus.cc:388] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:31:53.497395 5055 raft_consensus.cc:482] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:31:53.497423 5055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:53.498013 5055 raft_consensus.cc:504] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71c6b1ca3414a1ab943e628d60a5846" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38289 } }
I20260618 23:31:53.498140 5055 leader_election.cc:302] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a71c6b1ca3414a1ab943e628d60a5846; no voters:
I20260618 23:31:53.498322 5055 leader_election.cc:288] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:31:53.498383 5060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:53.498507 5060 raft_consensus.cc:686] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846 [term 1 LEADER]: Becoming Leader. State: Replica: a71c6b1ca3414a1ab943e628d60a5846, State: Running, Role: LEADER
I20260618 23:31:53.498667 5055 sys_catalog.cc:564] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:31:53.499073 5060 consensus_queue.cc:237] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71c6b1ca3414a1ab943e628d60a5846" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38289 } }
I20260618 23:31:53.499534 5060 sys_catalog.cc:455] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a71c6b1ca3414a1ab943e628d60a5846. Latest consensus state: current_term: 1 leader_uuid: "a71c6b1ca3414a1ab943e628d60a5846" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71c6b1ca3414a1ab943e628d60a5846" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38289 } } }
I20260618 23:31:53.499485 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.493556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38360 (local address 127.0.98.62:38289)
0618 23:31:53.493821 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:53.493830 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:53.493859 (+ 29us) server_negotiation.cc:407] Connection header received
0618 23:31:53.494607 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:53.494614 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:53.494658 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:53.494819 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:53.498698 (+ 3879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:53.498703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:53.498703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:31:53.498709 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:53.498777 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:31:53.498795 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:53.498853 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:53.498965 (+ 112us) server_negotiation.cc:299] Negotiation successful
0618 23:31:53.499233 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":120,"threads_started":1}
I20260618 23:31:53.499792 5060 sys_catalog.cc:458] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846 [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:53.499569 5062 sys_catalog.cc:455] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a71c6b1ca3414a1ab943e628d60a5846" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a71c6b1ca3414a1ab943e628d60a5846" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38289 } } }
I20260618 23:31:53.500283 5062 sys_catalog.cc:458] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846 [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:53.500109 5071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:31:53.501473 5071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:31:53.503273 5071 catalog_manager.cc:1347] Generated new cluster ID: e32cd564d9ac47c1bac39870e67834cd
I20260618 23:31:53.503325 5071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:31:53.522361 5071 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:31:53.522580 5071 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:31:53.531940 5071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846: Generated new TSK 0
I20260618 23:31:53.532153 5071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:31:53.538168 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38289
--builtin_ntp_servers=127.0.98.20:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260618 23:31:53.669728 5079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:53.670063 5079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:53.670101 5079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:53.672016 5079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:53.672082 5079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:53.672125 5079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:31:53.674219 5079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:53.674507 5079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:53.674937 5079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:53.681037 5084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:53.681213 5079 server_base.cc:1034] running on GCE node
W20260618 23:31:53.681037 5087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:53.681070 5085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:53.681584 5079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:53.682159 5079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:53.683462 5079 hybrid_clock.cc:648] HybridClock initialized: now 1781825513683436 us; error 45 us; skew 500 ppm
I20260618 23:31:53.685176 5079 webserver.cc:460] Webserver started at http://127.0.98.1:46011/ using document root <none> and password file <none>
I20260618 23:31:53.685487 5079 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:53.685568 5079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:53.685746 5079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:53.687237 5079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "04a403ffb90843aeb7004a66aeec9492"
format_stamp: "Formatted at 2026-06-18 23:31:53 on dist-test-slave-6224"
server_key: "440bcedbda255b4dca03d8b21b0c59d0"
server_key_iv: "59c5980e5f7d0da36e0f9f90cb035c7f"
server_key_version: "encryptionkey@0"
I20260618 23:31:53.687716 5079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "04a403ffb90843aeb7004a66aeec9492"
format_stamp: "Formatted at 2026-06-18 23:31:53 on dist-test-slave-6224"
server_key: "440bcedbda255b4dca03d8b21b0c59d0"
server_key_iv: "59c5980e5f7d0da36e0f9f90cb035c7f"
server_key_version: "encryptionkey@0"
I20260618 23:31:53.689522 5079 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260618 23:31:53.690598 5093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:53.690874 5079 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:53.691015 5079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "04a403ffb90843aeb7004a66aeec9492"
format_stamp: "Formatted at 2026-06-18 23:31:53 on dist-test-slave-6224"
server_key: "440bcedbda255b4dca03d8b21b0c59d0"
server_key_iv: "59c5980e5f7d0da36e0f9f90cb035c7f"
server_key_version: "encryptionkey@0"
I20260618 23:31:53.691123 5079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:53.695410 5079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:53.695822 5079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:53.696002 5079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:53.696297 5079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:53.696748 5079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:53.696830 5079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:53.696863 5079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:53.696883 5079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:53.706182 5079 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42439
I20260618 23:31:53.706296 5206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42439 every 8 connection(s)
I20260618 23:31:53.706676 5079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:31:53.710039 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.707521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50671 (local address 127.0.98.62:38289)
0618 23:31:53.707601 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:53.707602 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:53.707951 (+ 349us) server_negotiation.cc:407] Connection header received
0618 23:31:53.708464 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:53.708465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:53.708473 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:53.708530 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:53.709768 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:53.709769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:53.709770 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:31:53.709774 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:53.709790 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:31:53.709794 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:53.709852 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:53.709963 (+ 111us) server_negotiation.cc:299] Negotiation successful
0618 23:31:53.709978 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 23:31:53.710256 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.707518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38289 (local address 127.0.98.1:50671)
0618 23:31:53.707763 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:53.707800 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:53.708360 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:53.708598 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:53.708602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:53.708611 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:53.709609 (+ 998us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:31:53.709609 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:53.709640 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:31:53.709670 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:53.709885 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:53.709889 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:53.709891 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:31:53.709955 (+ 64us) client_negotiation.cc:240] Negotiation successful
0618 23:31:53.710008 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":139,"threads_started":1}
I20260618 23:31:53.710850 5207 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38289
I20260618 23:31:53.710935 5207 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:53.711117 5207 heartbeater.cc:511] Master 127.0.98.62:38289 requested a full tablet report, sending...
I20260618 23:31:53.711781 5019 ts_manager.cc:194] Registered new tserver with Master: 04a403ffb90843aeb7004a66aeec9492 (127.0.98.1:42439)
I20260618 23:31:53.716161 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 5079
I20260618 23:31:53.716259 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:31:53.716500 392 external_mini_cluster.cc:1428] Setting key 6e21e4f1f00f7167e029f298312673fa
I20260618 23:31:53.717712 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38289
--builtin_ntp_servers=127.0.98.20:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260618 23:31:53.807938 5210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:53.808169 5210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:53.808192 5210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:53.809401 5210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:53.809446 5210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:53.809473 5210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:31:53.810801 5210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:53.811075 5210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:53.811416 5210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:53.816962 5216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:53.816995 5215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:53.816962 5218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:53.817564 5210 server_base.cc:1034] running on GCE node
I20260618 23:31:53.817842 5210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:53.818123 5210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:53.819300 5210 hybrid_clock.cc:648] HybridClock initialized: now 1781825513819270 us; error 48 us; skew 500 ppm
I20260618 23:31:53.820780 5210 webserver.cc:460] Webserver started at http://127.0.98.2:44111/ using document root <none> and password file <none>
I20260618 23:31:53.821121 5210 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:53.821177 5210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:53.821316 5210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:53.822465 5210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "247d6359a630474f81265a6dcff5318a"
format_stamp: "Formatted at 2026-06-18 23:31:53 on dist-test-slave-6224"
server_key: "e6500f1b6f02c165d06bb819aa2d40be"
server_key_iv: "eceee557603a5fc7d6fc63e1af0c123f"
server_key_version: "encryptionkey@0"
I20260618 23:31:53.822857 5210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "247d6359a630474f81265a6dcff5318a"
format_stamp: "Formatted at 2026-06-18 23:31:53 on dist-test-slave-6224"
server_key: "e6500f1b6f02c165d06bb819aa2d40be"
server_key_iv: "eceee557603a5fc7d6fc63e1af0c123f"
server_key_version: "encryptionkey@0"
I20260618 23:31:53.824507 5210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:31:53.825522 5224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:53.825822 5210 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:31:53.825946 5210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "247d6359a630474f81265a6dcff5318a"
format_stamp: "Formatted at 2026-06-18 23:31:53 on dist-test-slave-6224"
server_key: "e6500f1b6f02c165d06bb819aa2d40be"
server_key_iv: "eceee557603a5fc7d6fc63e1af0c123f"
server_key_version: "encryptionkey@0"
I20260618 23:31:53.826026 5210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:53.829589 5210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:53.829890 5210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:53.830037 5210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:53.830253 5210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:53.830569 5210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:53.830624 5210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:53.830665 5210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:53.830694 5210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:53.837718 5210 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36819
I20260618 23:31:53.837767 5337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36819 every 8 connection(s)
I20260618 23:31:53.838120 5210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:31:53.841809 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.838918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60501 (local address 127.0.98.62:38289)
0618 23:31:53.839005 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:53.839007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:53.839267 (+ 260us) server_negotiation.cc:407] Connection header received
0618 23:31:53.840061 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:53.840062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:53.840070 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:53.840164 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:53.841452 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:53.841454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:53.841455 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:31:53.841460 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:53.841480 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:31:53.841485 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:53.841569 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:53.841710 (+ 141us) server_negotiation.cc:299] Negotiation successful
0618 23:31:53.841727 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 23:31:53.842067 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.838892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38289 (local address 127.0.98.2:60501)
0618 23:31:53.839084 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:53.839134 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:53.839759 (+ 625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:53.840240 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:53.840244 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:53.840252 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:53.841254 (+ 1002us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:31:53.841254 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:53.841285 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:31:53.841315 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:53.841603 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:53.841609 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:53.841611 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:31:53.841703 (+ 92us) client_negotiation.cc:240] Negotiation successful
0618 23:31:53.841759 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260618 23:31:53.842841 5338 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38289
I20260618 23:31:53.842984 5338 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:53.843175 5338 heartbeater.cc:511] Master 127.0.98.62:38289 requested a full tablet report, sending...
I20260618 23:31:53.843370 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 5210
I20260618 23:31:53.843520 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:31:53.843677 5019 ts_manager.cc:194] Registered new tserver with Master: 247d6359a630474f81265a6dcff5318a (127.0.98.2:36819)
I20260618 23:31:53.843801 392 external_mini_cluster.cc:1428] Setting key cc7a25314528eb4ffa41923380076a94
I20260618 23:31:53.844880 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38289
--builtin_ntp_servers=127.0.98.20:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260618 23:31:53.938913 5341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:53.939239 5341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:53.939281 5341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:53.940631 5341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:53.940706 5341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:53.940753 5341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:31:53.942210 5341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38289
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:53.942523 5341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:53.942876 5341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:53.948726 5346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:53.948983 5341 server_base.cc:1034] running on GCE node
W20260618 23:31:53.948710 5349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:53.948707 5347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:53.949402 5341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:53.949703 5341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:53.950920 5341 hybrid_clock.cc:648] HybridClock initialized: now 1781825513950906 us; error 62 us; skew 500 ppm
I20260618 23:31:53.952610 5341 webserver.cc:460] Webserver started at http://127.0.98.3:45365/ using document root <none> and password file <none>
I20260618 23:31:53.952925 5341 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:53.953006 5341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:53.953200 5341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:53.954461 5341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "521978bca5c64ed6a2ec987cf4f7d899"
format_stamp: "Formatted at 2026-06-18 23:31:53 on dist-test-slave-6224"
server_key: "52912d8cfce1b694bb8d484e1422ff0c"
server_key_iv: "09d919cb8f83730f02c297ee34c8045d"
server_key_version: "encryptionkey@0"
I20260618 23:31:53.954877 5341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "521978bca5c64ed6a2ec987cf4f7d899"
format_stamp: "Formatted at 2026-06-18 23:31:53 on dist-test-slave-6224"
server_key: "52912d8cfce1b694bb8d484e1422ff0c"
server_key_iv: "09d919cb8f83730f02c297ee34c8045d"
server_key_version: "encryptionkey@0"
I20260618 23:31:53.956648 5341 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260618 23:31:53.957882 5355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:53.958280 5341 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:31:53.958467 5341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "521978bca5c64ed6a2ec987cf4f7d899"
format_stamp: "Formatted at 2026-06-18 23:31:53 on dist-test-slave-6224"
server_key: "52912d8cfce1b694bb8d484e1422ff0c"
server_key_iv: "09d919cb8f83730f02c297ee34c8045d"
server_key_version: "encryptionkey@0"
I20260618 23:31:53.958573 5341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:53.963690 5341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:53.964076 5341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:53.964257 5341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:53.964512 5341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:53.964895 5341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:53.964951 5341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:53.964996 5341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:53.965025 5341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:53.973856 5341 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39767
I20260618 23:31:53.973958 5468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39767 every 8 connection(s)
I20260618 23:31:53.974398 5341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:31:53.978359 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.975464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55939 (local address 127.0.98.62:38289)
0618 23:31:53.975638 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:53.975641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:53.976001 (+ 360us) server_negotiation.cc:407] Connection header received
0618 23:31:53.976539 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:53.976540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:53.976548 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:53.976617 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:53.977987 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:53.977989 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:53.977989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:31:53.977993 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:53.978010 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:31:53.978014 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:53.978073 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:53.978276 (+ 203us) server_negotiation.cc:299] Negotiation successful
0618 23:31:53.978293 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260618 23:31:53.978605 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.975466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38289 (local address 127.0.98.3:55939)
0618 23:31:53.975832 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:53.975875 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:53.976426 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:53.976685 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:53.976690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:53.976700 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:53.977808 (+ 1108us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:31:53.977809 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:53.977843 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:31:53.977875 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:53.978164 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:53.978172 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:53.978175 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:31:53.978277 (+ 102us) client_negotiation.cc:240] Negotiation successful
0618 23:31:53.978348 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":173,"threads_started":1}
I20260618 23:31:53.979408 5469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38289
I20260618 23:31:53.979508 5469 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:53.979700 5469 heartbeater.cc:511] Master 127.0.98.62:38289 requested a full tablet report, sending...
I20260618 23:31:53.980091 5019 ts_manager.cc:194] Registered new tserver with Master: 521978bca5c64ed6a2ec987cf4f7d899 (127.0.98.3:39767)
I20260618 23:31:53.981122 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 5341
I20260618 23:31:53.981292 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:31:53.981516 392 external_mini_cluster.cc:1428] Setting key 78bb07a6d6cb9cbe91a762643e08d526
I20260618 23:31:53.982627 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:31:53.985574 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.984211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38374 (local address 127.0.98.62:38289)
0618 23:31:53.984294 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:53.984296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:53.984419 (+ 123us) server_negotiation.cc:407] Connection header received
0618 23:31:53.984469 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:53.984469 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:53.984475 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:53.984541 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:53.985288 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:53.985289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:53.985289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:31:53.985293 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:53.985309 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:31:53.985335 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:53.985392 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:53.985495 (+ 103us) server_negotiation.cc:299] Negotiation successful
0618 23:31:53.985507 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 23:31:53.987681 5019 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:31:53.989531 5019 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:31:53.994551 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.992419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39767 (local address 127.0.0.1:50998)
0618 23:31:53.992626 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:53.992712 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:53.992784 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:53.993224 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:53.993230 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:53.993237 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:53.993904 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:31:53.993904 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:53.993922 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:31:53.993971 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:53.994338 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:53.994344 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:53.994347 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:31:53.994423 (+ 76us) client_negotiation.cc:240] Negotiation successful
0618 23:31:53.994442 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":75,"threads_started":1}
I20260618 23:31:53.994588 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.992732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.0.98.3:39767)
0618 23:31:53.993008 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:53.993014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:53.993026 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:31:53.993053 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:53.993056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:53.993068 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:53.993158 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:53.994124 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:53.994138 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:53.994138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:31:53.994144 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:53.994223 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:31:53.994265 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:53.994347 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:53.994469 (+ 122us) server_negotiation.cc:299] Negotiation successful
0618 23:31:53.994491 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":92,"threads_started":1}
I20260618 23:31:53.994707 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.992527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46750 (local address 127.0.98.2:36819)
0618 23:31:53.993088 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:53.993094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:53.993105 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:31:53.993135 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:53.993138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:53.993149 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:53.993234 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:53.994230 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:53.994242 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:53.994243 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:31:53.994249 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:53.994296 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:31:53.994338 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:53.994410 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:53.994591 (+ 181us) server_negotiation.cc:299] Negotiation successful
0618 23:31:53.994609 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":74,"threads_started":1}
I20260618 23:31:53.994725 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.992419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36819 (local address 127.0.0.1:46750)
0618 23:31:53.992693 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:53.992712 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:53.992784 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:53.993608 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:53.993609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:53.993612 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:53.994096 (+ 484us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:31:53.994096 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:53.994103 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:31:53.994115 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:53.994445 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:53.994446 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:53.994448 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:31:53.994498 (+ 50us) client_negotiation.cc:240] Negotiation successful
0618 23:31:53.994516 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":5,"thread_start_us":71,"threads_started":1}
I20260618 23:31:53.995486 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.992463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42439 (local address 127.0.0.1:57192)
0618 23:31:53.992889 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:53.992905 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:53.992938 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:53.994109 (+ 1171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:53.994110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:53.994114 (+ 4us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:53.995099 (+ 985us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:31:53.995099 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:53.995109 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:31:53.995124 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:53.995348 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:53.995349 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:53.995350 (+ 1us) client_negotiation.cc:769] Sending connection context
0618 23:31:53.995395 (+ 45us) client_negotiation.cc:240] Negotiation successful
0618 23:31:53.995412 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":135,"threads_started":1}
I20260618 23:31:53.995775 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:53.992747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57192 (local address 127.0.98.1:42439)
0618 23:31:53.993652 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:53.993657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:53.993668 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:31:53.993697 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:53.993699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:53.993709 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:53.993788 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:53.995190 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:53.995195 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:53.995195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:31:53.995200 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:53.995233 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:31:53.995262 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:53.995322 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:53.995680 (+ 358us) server_negotiation.cc:299] Negotiation successful
0618 23:31:53.995698 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":866,"thread_start_us":84,"threads_started":1}
I20260618 23:31:53.995817 5403 tablet_service.cc:1467] Processing CreateTablet for tablet ea392327731f41a587d539f78c709273 (DEFAULT_TABLE table=test-table [id=5a59f110e2c6463bb386cd6619801e31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:53.996026 5272 tablet_service.cc:1467] Processing CreateTablet for tablet ea392327731f41a587d539f78c709273 (DEFAULT_TABLE table=test-table [id=5a59f110e2c6463bb386cd6619801e31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:53.996275 5403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea392327731f41a587d539f78c709273. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:53.996435 5272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea392327731f41a587d539f78c709273. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:53.996501 5141 tablet_service.cc:1467] Processing CreateTablet for tablet ea392327731f41a587d539f78c709273 (DEFAULT_TABLE table=test-table [id=5a59f110e2c6463bb386cd6619801e31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:31:53.996816 5141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea392327731f41a587d539f78c709273. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:53.999706 5483 tablet_bootstrap.cc:492] T ea392327731f41a587d539f78c709273 P 521978bca5c64ed6a2ec987cf4f7d899: Bootstrap starting.
I20260618 23:31:54.000674 5485 tablet_bootstrap.cc:492] T ea392327731f41a587d539f78c709273 P 247d6359a630474f81265a6dcff5318a: Bootstrap starting.
I20260618 23:31:54.000777 5483 tablet_bootstrap.cc:654] T ea392327731f41a587d539f78c709273 P 521978bca5c64ed6a2ec987cf4f7d899: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:54.001148 5483 log.cc:824] T ea392327731f41a587d539f78c709273 P 521978bca5c64ed6a2ec987cf4f7d899: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:54.001459 5485 tablet_bootstrap.cc:654] T ea392327731f41a587d539f78c709273 P 247d6359a630474f81265a6dcff5318a: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:54.001765 5485 log.cc:824] T ea392327731f41a587d539f78c709273 P 247d6359a630474f81265a6dcff5318a: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:54.001951 5483 tablet_bootstrap.cc:492] T ea392327731f41a587d539f78c709273 P 521978bca5c64ed6a2ec987cf4f7d899: No bootstrap required, opened a new log
I20260618 23:31:54.002041 5483 ts_tablet_manager.cc:1397] T ea392327731f41a587d539f78c709273 P 521978bca5c64ed6a2ec987cf4f7d899: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:31:54.002123 5484 tablet_bootstrap.cc:492] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492: Bootstrap starting.
I20260618 23:31:54.002702 5484 tablet_bootstrap.cc:654] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:54.003008 5484 log.cc:824] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:54.003651 5485 tablet_bootstrap.cc:492] T ea392327731f41a587d539f78c709273 P 247d6359a630474f81265a6dcff5318a: No bootstrap required, opened a new log
I20260618 23:31:54.003736 5485 ts_tablet_manager.cc:1397] T ea392327731f41a587d539f78c709273 P 247d6359a630474f81265a6dcff5318a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260618 23:31:54.003945 5484 tablet_bootstrap.cc:492] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492: No bootstrap required, opened a new log
I20260618 23:31:54.004052 5484 ts_tablet_manager.cc:1397] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:31:54.005064 5483 raft_consensus.cc:348] T ea392327731f41a587d539f78c709273 P 521978bca5c64ed6a2ec987cf4f7d899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521978bca5c64ed6a2ec987cf4f7d899" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39767 } } peers { permanent_uuid: "247d6359a630474f81265a6dcff5318a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36819 } } peers { permanent_uuid: "04a403ffb90843aeb7004a66aeec9492" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42439 } }
I20260618 23:31:54.005292 5483 raft_consensus.cc:374] T ea392327731f41a587d539f78c709273 P 521978bca5c64ed6a2ec987cf4f7d899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:54.005332 5483 raft_consensus.cc:729] T ea392327731f41a587d539f78c709273 P 521978bca5c64ed6a2ec987cf4f7d899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521978bca5c64ed6a2ec987cf4f7d899, State: Initialized, Role: FOLLOWER
I20260618 23:31:54.005501 5483 consensus_queue.cc:260] T ea392327731f41a587d539f78c709273 P 521978bca5c64ed6a2ec987cf4f7d899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521978bca5c64ed6a2ec987cf4f7d899" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39767 } } peers { permanent_uuid: "247d6359a630474f81265a6dcff5318a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36819 } } peers { permanent_uuid: "04a403ffb90843aeb7004a66aeec9492" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42439 } }
I20260618 23:31:54.005833 5483 ts_tablet_manager.cc:1428] T ea392327731f41a587d539f78c709273 P 521978bca5c64ed6a2ec987cf4f7d899: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260618 23:31:54.005895 5469 heartbeater.cc:503] Master 127.0.98.62:38289 was elected leader, sending a full tablet report...
I20260618 23:31:54.005981 5485 raft_consensus.cc:348] T ea392327731f41a587d539f78c709273 P 247d6359a630474f81265a6dcff5318a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521978bca5c64ed6a2ec987cf4f7d899" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39767 } } peers { permanent_uuid: "247d6359a630474f81265a6dcff5318a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36819 } } peers { permanent_uuid: "04a403ffb90843aeb7004a66aeec9492" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42439 } }
I20260618 23:31:54.006145 5485 raft_consensus.cc:374] T ea392327731f41a587d539f78c709273 P 247d6359a630474f81265a6dcff5318a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:54.006175 5485 raft_consensus.cc:729] T ea392327731f41a587d539f78c709273 P 247d6359a630474f81265a6dcff5318a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247d6359a630474f81265a6dcff5318a, State: Initialized, Role: FOLLOWER
I20260618 23:31:54.006330 5485 consensus_queue.cc:260] T ea392327731f41a587d539f78c709273 P 247d6359a630474f81265a6dcff5318a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521978bca5c64ed6a2ec987cf4f7d899" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39767 } } peers { permanent_uuid: "247d6359a630474f81265a6dcff5318a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36819 } } peers { permanent_uuid: "04a403ffb90843aeb7004a66aeec9492" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42439 } }
I20260618 23:31:54.006670 5338 heartbeater.cc:503] Master 127.0.98.62:38289 was elected leader, sending a full tablet report...
I20260618 23:31:54.006667 5485 ts_tablet_manager.cc:1428] T ea392327731f41a587d539f78c709273 P 247d6359a630474f81265a6dcff5318a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260618 23:31:54.007014 5484 raft_consensus.cc:348] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521978bca5c64ed6a2ec987cf4f7d899" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39767 } } peers { permanent_uuid: "247d6359a630474f81265a6dcff5318a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36819 } } peers { permanent_uuid: "04a403ffb90843aeb7004a66aeec9492" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42439 } }
I20260618 23:31:54.007187 5484 raft_consensus.cc:374] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:54.007228 5484 raft_consensus.cc:729] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a403ffb90843aeb7004a66aeec9492, State: Initialized, Role: FOLLOWER
I20260618 23:31:54.007350 5484 consensus_queue.cc:260] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521978bca5c64ed6a2ec987cf4f7d899" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39767 } } peers { permanent_uuid: "247d6359a630474f81265a6dcff5318a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36819 } } peers { permanent_uuid: "04a403ffb90843aeb7004a66aeec9492" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42439 } }
I20260618 23:31:54.007671 5484 ts_tablet_manager.cc:1428] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260618 23:31:54.007716 5207 heartbeater.cc:503] Master 127.0.98.62:38289 was elected leader, sending a full tablet report...
I20260618 23:31:54.086845 5491 raft_consensus.cc:482] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:31:54.087034 5491 raft_consensus.cc:504] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521978bca5c64ed6a2ec987cf4f7d899" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39767 } } peers { permanent_uuid: "247d6359a630474f81265a6dcff5318a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36819 } } peers { permanent_uuid: "04a403ffb90843aeb7004a66aeec9492" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42439 } }
I20260618 23:31:54.087632 5491 leader_election.cc:288] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 521978bca5c64ed6a2ec987cf4f7d899 (127.0.98.3:39767), 247d6359a630474f81265a6dcff5318a (127.0.98.2:36819)
W20260618 23:31:54.088711 5339 tablet.cc:2369] T ea392327731f41a587d539f78c709273 P 247d6359a630474f81265a6dcff5318a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:31:54.089177 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:54.087952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59479 (local address 127.0.98.3:39767)
0618 23:31:54.088035 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:54.088037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:54.088046 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:31:54.088076 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:54.088077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:54.088084 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:54.088150 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:54.088866 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:54.088868 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:54.088868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:31:54.088875 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:54.088898 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:31:54.088904 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:54.088961 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:54.089055 (+ 94us) server_negotiation.cc:299] Negotiation successful
0618 23:31:54.089077 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 23:31:54.089169 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:54.087835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39767 (local address 127.0.98.1:59479)
0618 23:31:54.087895 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:54.087912 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:54.087964 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:54.088197 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:54.088199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:54.088202 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:54.088725 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:31:54.088726 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:54.088736 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:31:54.088753 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:54.088987 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:54.088988 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:54.088988 (+ 0us) client_negotiation.cc:769] Sending connection context
0618 23:31:54.089044 (+ 56us) client_negotiation.cc:240] Negotiation successful
0618 23:31:54.089062 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260618 23:31:54.089452 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:54.088004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40641 (local address 127.0.98.2:36819)
0618 23:31:54.088060 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:54.088063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:54.088237 (+ 174us) server_negotiation.cc:407] Connection header received
0618 23:31:54.088298 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:54.088299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:54.088307 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:54.088357 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:54.089157 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:54.089158 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:54.089159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:31:54.089166 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:54.089187 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:31:54.089191 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:54.089255 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:54.089376 (+ 121us) server_negotiation.cc:299] Negotiation successful
0618 23:31:54.089393 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260618 23:31:54.089457 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:54.087936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36819 (local address 127.0.98.1:40641)
0618 23:31:54.088140 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:54.088149 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:54.088235 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:54.088360 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:54.088361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:54.088364 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:54.088975 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:31:54.088976 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:54.088985 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:31:54.089002 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:54.089301 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:54.089302 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:54.089303 (+ 1us) client_negotiation.cc:769] Sending connection context
0618 23:31:54.089369 (+ 66us) client_negotiation.cc:240] Negotiation successful
0618 23:31:54.089383 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":136,"threads_started":1}
I20260618 23:31:54.089603 5423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea392327731f41a587d539f78c709273" candidate_uuid: "04a403ffb90843aeb7004a66aeec9492" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521978bca5c64ed6a2ec987cf4f7d899" is_pre_election: true
I20260618 23:31:54.089795 5423 raft_consensus.cc:2454] T ea392327731f41a587d539f78c709273 P 521978bca5c64ed6a2ec987cf4f7d899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04a403ffb90843aeb7004a66aeec9492 in term 0.
I20260618 23:31:54.089847 5292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea392327731f41a587d539f78c709273" candidate_uuid: "04a403ffb90843aeb7004a66aeec9492" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247d6359a630474f81265a6dcff5318a" is_pre_election: true
I20260618 23:31:54.090023 5292 raft_consensus.cc:2454] T ea392327731f41a587d539f78c709273 P 247d6359a630474f81265a6dcff5318a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04a403ffb90843aeb7004a66aeec9492 in term 0.
I20260618 23:31:54.090075 5096 leader_election.cc:302] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a403ffb90843aeb7004a66aeec9492, 521978bca5c64ed6a2ec987cf4f7d899; no voters:
I20260618 23:31:54.090247 5491 raft_consensus.cc:2781] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:31:54.090312 5491 raft_consensus.cc:482] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:31:54.090348 5491 raft_consensus.cc:3037] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:54.091073 5491 raft_consensus.cc:504] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521978bca5c64ed6a2ec987cf4f7d899" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39767 } } peers { permanent_uuid: "247d6359a630474f81265a6dcff5318a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36819 } } peers { permanent_uuid: "04a403ffb90843aeb7004a66aeec9492" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42439 } }
I20260618 23:31:54.091234 5491 leader_election.cc:288] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492 [CANDIDATE]: Term 1 election: Requested vote from peers 521978bca5c64ed6a2ec987cf4f7d899 (127.0.98.3:39767), 247d6359a630474f81265a6dcff5318a (127.0.98.2:36819)
I20260618 23:31:54.091415 5423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea392327731f41a587d539f78c709273" candidate_uuid: "04a403ffb90843aeb7004a66aeec9492" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521978bca5c64ed6a2ec987cf4f7d899"
I20260618 23:31:54.091409 5292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea392327731f41a587d539f78c709273" candidate_uuid: "04a403ffb90843aeb7004a66aeec9492" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "247d6359a630474f81265a6dcff5318a"
I20260618 23:31:54.091499 5423 raft_consensus.cc:3037] T ea392327731f41a587d539f78c709273 P 521978bca5c64ed6a2ec987cf4f7d899 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:54.091531 5292 raft_consensus.cc:3037] T ea392327731f41a587d539f78c709273 P 247d6359a630474f81265a6dcff5318a [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:54.092357 5423 raft_consensus.cc:2454] T ea392327731f41a587d539f78c709273 P 521978bca5c64ed6a2ec987cf4f7d899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04a403ffb90843aeb7004a66aeec9492 in term 1.
I20260618 23:31:54.092497 5292 raft_consensus.cc:2454] T ea392327731f41a587d539f78c709273 P 247d6359a630474f81265a6dcff5318a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04a403ffb90843aeb7004a66aeec9492 in term 1.
I20260618 23:31:54.092528 5096 leader_election.cc:302] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04a403ffb90843aeb7004a66aeec9492, 521978bca5c64ed6a2ec987cf4f7d899; no voters:
I20260618 23:31:54.092628 5491 raft_consensus.cc:2781] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:54.092779 5491 raft_consensus.cc:686] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492 [term 1 LEADER]: Becoming Leader. State: Replica: 04a403ffb90843aeb7004a66aeec9492, State: Running, Role: LEADER
I20260618 23:31:54.092876 5491 consensus_queue.cc:237] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521978bca5c64ed6a2ec987cf4f7d899" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39767 } } peers { permanent_uuid: "247d6359a630474f81265a6dcff5318a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36819 } } peers { permanent_uuid: "04a403ffb90843aeb7004a66aeec9492" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42439 } }
I20260618 23:31:54.093866 5019 catalog_manager.cc:5495] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04a403ffb90843aeb7004a66aeec9492 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "04a403ffb90843aeb7004a66aeec9492" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521978bca5c64ed6a2ec987cf4f7d899" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "247d6359a630474f81265a6dcff5318a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36819 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04a403ffb90843aeb7004a66aeec9492" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42439 } health_report { overall_health: HEALTHY } } }
I20260618 23:31:54.112881 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:54.111675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57202 (local address 127.0.98.1:42439)
0618 23:31:54.111755 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:54.111756 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:54.111766 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:31:54.111809 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:54.111810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:54.111830 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:54.111899 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:54.112633 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:54.112634 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:54.112634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:31:54.112638 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:54.112653 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:31:54.112657 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:54.112721 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:54.112796 (+ 75us) server_negotiation.cc:299] Negotiation successful
0618 23:31:54.112809 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 23:31:54.115258 5423 raft_consensus.cc:1261] T ea392327731f41a587d539f78c709273 P 521978bca5c64ed6a2ec987cf4f7d899 [term 1 FOLLOWER]: Refusing update from remote peer 04a403ffb90843aeb7004a66aeec9492: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:31:54.115294 5292 raft_consensus.cc:1261] T ea392327731f41a587d539f78c709273 P 247d6359a630474f81265a6dcff5318a [term 1 FOLLOWER]: Refusing update from remote peer 04a403ffb90843aeb7004a66aeec9492: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:31:54.115804 5491 consensus_queue.cc:1035] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521978bca5c64ed6a2ec987cf4f7d899" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:31:54.115980 5493 consensus_queue.cc:1035] T ea392327731f41a587d539f78c709273 P 04a403ffb90843aeb7004a66aeec9492 [LEADER]: Connected to new peer: Peer: permanent_uuid: "247d6359a630474f81265a6dcff5318a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:31:54.118837 5498 mvcc.cc:205] Tried to move back new op lower bound from 7298357305812082688 to 7298357305725038592. Current Snapshot: MvccSnapshot[applied={T|T < 7298357305812082688}]
I20260618 23:31:54.118979 5497 mvcc.cc:205] Tried to move back new op lower bound from 7298357305812082688 to 7298357305725038592. Current Snapshot: MvccSnapshot[applied={T|T < 7298357305812082688}]
I20260618 23:31:54.121622 5019 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:31:54.121716 5019 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:31:54.122681 5019 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a71c6b1ca3414a1ab943e628d60a5846: Sending DeleteTablet for 3 replicas of tablet ea392327731f41a587d539f78c709273
I20260618 23:31:54.123072 5141 tablet_service.cc:1514] Processing DeleteTablet for tablet ea392327731f41a587d539f78c709273 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:54 UTC) from {username='slave'} at 127.0.0.1:57192
I20260618 23:31:54.123219 5403 tablet_service.cc:1514] Processing DeleteTablet for tablet ea392327731f41a587d539f78c709273 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:54 UTC) from {username='slave'} at 127.0.0.1:50998
I20260618 23:31:54.123234 5272 tablet_service.cc:1514] Processing DeleteTablet for tablet ea392327731f41a587d539f78c709273 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:31:54 UTC) from {username='slave'} at 127.0.0.1:46750
I20260618 23:31:54.123486 5507 tablet_replica.cc:330] T ea392327731f41a587d539f78c709273 P 247d6359a630474f81265a6dcff5318a: stopping tablet replica
I20260618 23:31:54.123624 5507 raft_consensus.cc:2229] T ea392327731f41a587d539f78c709273 P 247d6359a630474f81265a6dcff5318a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:31:54.123761 5507 raft_consensus.cc:2258] T ea392327731f41a587d539f78c709273 P 247d6359a630474f81265a6dcff5318a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:54.123881 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 5079
I20260618 23:31:54.123883 5508 tablet_replica.cc:330] T ea392327731f41a587d539f78c709273 P 521978bca5c64ed6a2ec987cf4f7d899: stopping tablet replica
I20260618 23:31:54.123986 5508 raft_consensus.cc:2229] T ea392327731f41a587d539f78c709273 P 521978bca5c64ed6a2ec987cf4f7d899 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:31:54.124091 5508 raft_consensus.cc:2258] T ea392327731f41a587d539f78c709273 P 521978bca5c64ed6a2ec987cf4f7d899 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:54.124708 5508 ts_tablet_manager.cc:1905] T ea392327731f41a587d539f78c709273 P 521978bca5c64ed6a2ec987cf4f7d899: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:31:54.126034 5507 ts_tablet_manager.cc:1905] T ea392327731f41a587d539f78c709273 P 247d6359a630474f81265a6dcff5318a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:31:54.126904 5508 ts_tablet_manager.cc:1918] T ea392327731f41a587d539f78c709273 P 521978bca5c64ed6a2ec987cf4f7d899: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:31:54.126986 5508 log.cc:1196] T ea392327731f41a587d539f78c709273 P 521978bca5c64ed6a2ec987cf4f7d899: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/ea392327731f41a587d539f78c709273
I20260618 23:31:54.127274 5508 ts_tablet_manager.cc:1939] T ea392327731f41a587d539f78c709273 P 521978bca5c64ed6a2ec987cf4f7d899: Deleting consensus metadata
I20260618 23:31:54.127490 5507 ts_tablet_manager.cc:1918] T ea392327731f41a587d539f78c709273 P 247d6359a630474f81265a6dcff5318a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:31:54.127602 5507 log.cc:1196] T ea392327731f41a587d539f78c709273 P 247d6359a630474f81265a6dcff5318a: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/ea392327731f41a587d539f78c709273
I20260618 23:31:54.127750 5005 catalog_manager.cc:4841] TS 521978bca5c64ed6a2ec987cf4f7d899 (127.0.98.3:39767): tablet ea392327731f41a587d539f78c709273 (table test-table [id=5a59f110e2c6463bb386cd6619801e31]) successfully deleted
I20260618 23:31:54.127898 5507 ts_tablet_manager.cc:1939] T ea392327731f41a587d539f78c709273 P 247d6359a630474f81265a6dcff5318a: Deleting consensus metadata
I20260618 23:31:54.128381 5003 catalog_manager.cc:4841] TS 247d6359a630474f81265a6dcff5318a (127.0.98.2:36819): tablet ea392327731f41a587d539f78c709273 (table test-table [id=5a59f110e2c6463bb386cd6619801e31]) successfully deleted
W20260618 23:31:54.131096 5004 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42439 (error 108)
I20260618 23:31:54.131829 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:54.131635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42439 (local address 127.0.0.1:57208)
0618 23:31:54.131710 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:54.131780 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260618 23:31:54.131824 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 5210
W20260618 23:31:54.133222 5004 catalog_manager.cc:4584] TS 04a403ffb90843aeb7004a66aeec9492 (127.0.98.1:42439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ea392327731f41a587d539f78c709273: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42439: connect: Connection refused (error 111)
I20260618 23:31:54.137579 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 5341
I20260618 23:31:54.142907 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 4988
2026-06-18T23:31:54Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (795 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260618 23:31:54.164453 392 mini_oidc.cc:152] Starting JWKS server
I20260618 23:31:54.165637 392 webserver.cc:460] Webserver started at https://127.0.0.1:46263/ using document root <none> and password file <none>
I20260618 23:31:54.165786 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260618 23:31:54.165993 392 webserver.cc:460] Webserver started at http://127.0.0.1:41543/ using document root <none> and password file <none>
2026-06-18T23:31:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:31:54Z Disabled control of system clock
I20260618 23:31:54.175275 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42203
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42203
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46263/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825489608381-392-0/testchainca.cert with env {}
W20260618 23:31:54.263599 5515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:54.263837 5515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:54.263861 5515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:54.265148 5515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:31:54.265206 5515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:54.265220 5515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:54.265233 5515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:31:54.265246 5515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260618 23:31:54.265259 5515 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260618 23:31:54.265271 5515 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46263/jwks.json
I20260618 23:31:54.266928 5515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42203
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46263/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825489608381-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:54.267207 5515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:54.267448 5515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:54.272908 5520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:54.272858 5523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:54.272867 5521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:54.273146 5515 server_base.cc:1034] running on GCE node
I20260618 23:31:54.273437 5515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:54.273751 5515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:54.274938 5515 hybrid_clock.cc:648] HybridClock initialized: now 1781825514274910 us; error 41 us; skew 500 ppm
I20260618 23:31:54.276525 5515 webserver.cc:460] Webserver started at http://127.0.98.62:39659/ using document root <none> and password file <none>
I20260618 23:31:54.276757 5515 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:54.276819 5515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:54.276926 5515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:54.278151 5515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "6f8c6562f6c448638ad5a1368f9271d5"
format_stamp: "Formatted at 2026-06-18 23:31:54 on dist-test-slave-6224"
server_key: "cf83eca0aa96c4b931892f7a6b3e9970"
server_key_iv: "c92803ac2268704deab860dad48e188d"
server_key_version: "encryptionkey@0"
I20260618 23:31:54.278498 5515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "6f8c6562f6c448638ad5a1368f9271d5"
format_stamp: "Formatted at 2026-06-18 23:31:54 on dist-test-slave-6224"
server_key: "cf83eca0aa96c4b931892f7a6b3e9970"
server_key_iv: "c92803ac2268704deab860dad48e188d"
server_key_version: "encryptionkey@0"
I20260618 23:31:54.280064 5515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:31:54.281055 5529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:54.281313 5515 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:54.281448 5515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "6f8c6562f6c448638ad5a1368f9271d5"
format_stamp: "Formatted at 2026-06-18 23:31:54 on dist-test-slave-6224"
server_key: "cf83eca0aa96c4b931892f7a6b3e9970"
server_key_iv: "c92803ac2268704deab860dad48e188d"
server_key_version: "encryptionkey@0"
I20260618 23:31:54.281556 5515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:54.305400 5515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:54.305773 5515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:54.305948 5515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:54.310529 5515 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42203
I20260618 23:31:54.310671 5581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42203 every 8 connection(s)
I20260618 23:31:54.310990 5515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:31:54.312245 5582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:54.315112 5582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5: Bootstrap starting.
I20260618 23:31:54.315939 5582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:54.316293 5582 log.cc:824] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:54.317189 5582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5: No bootstrap required, opened a new log
I20260618 23:31:54.318980 5582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8c6562f6c448638ad5a1368f9271d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42203 } }
I20260618 23:31:54.319166 5582 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:54.319209 5582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f8c6562f6c448638ad5a1368f9271d5, State: Initialized, Role: FOLLOWER
I20260618 23:31:54.319419 5582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8c6562f6c448638ad5a1368f9271d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42203 } }
I20260618 23:31:54.319530 5582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:31:54.319617 5582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:31:54.319666 5582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:54.320253 5582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8c6562f6c448638ad5a1368f9271d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42203 } }
I20260618 23:31:54.320355 5582 leader_election.cc:302] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f8c6562f6c448638ad5a1368f9271d5; no voters:
I20260618 23:31:54.320518 5582 leader_election.cc:288] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:31:54.320690 5585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:54.320845 5582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:31:54.320973 5585 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5 [term 1 LEADER]: Becoming Leader. State: Replica: 6f8c6562f6c448638ad5a1368f9271d5, State: Running, Role: LEADER
I20260618 23:31:54.321103 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 5515
I20260618 23:31:54.321115 5585 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8c6562f6c448638ad5a1368f9271d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42203 } }
I20260618 23:31:54.321208 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:31:54.321434 392 external_mini_cluster.cc:1428] Setting key e5a9c68a80bcee931ba305504114b35a
I20260618 23:31:54.321667 5586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f8c6562f6c448638ad5a1368f9271d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8c6562f6c448638ad5a1368f9271d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42203 } } }
I20260618 23:31:54.321750 5586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5 [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:54.322198 5587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f8c6562f6c448638ad5a1368f9271d5. Latest consensus state: current_term: 1 leader_uuid: "6f8c6562f6c448638ad5a1368f9271d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f8c6562f6c448638ad5a1368f9271d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42203 } } }
I20260618 23:31:54.322294 5587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5 [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:54.322559 5594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:31:54.323398 5594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:31:54.325448 5594 catalog_manager.cc:1347] Generated new cluster ID: 04cb3e7a4b8543f399f2ae4d30a29692
I20260618 23:31:54.325522 5594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:31:54.328447 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:54.322219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.0.98.62:42203)
0618 23:31:54.322577 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:54.322587 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:54.322851 (+ 264us) server_negotiation.cc:407] Connection header received
0618 23:31:54.323103 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:54.323110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:54.323174 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:54.323361 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:54.323665 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:54.324837 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:54.326096 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:54.326724 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:54.326841 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:54.327695 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:54.327699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:54.327699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:31:54.327918 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:54.327960 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:31:54.327975 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:54.328046 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:54.328147 (+ 101us) server_negotiation.cc:299] Negotiation successful
0618 23:31:54.328229 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":81,"threads_started":1}
I20260618 23:31:54.329722 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:31:54.341648 5594 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:31:54.342294 5594 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:31:54.361572 5594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f8c6562f6c448638ad5a1368f9271d5: Generated new TSK 0
I20260618 23:31:54.361889 5594 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260618 23:31:54.399106 5612 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260618 23:31:54.399222 5612 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260618 23:31:54.399590 5601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:54.377823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41454 (local address 127.0.98.62:42203)
0618 23:31:54.377942 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:54.377945 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:54.378180 (+ 235us) server_negotiation.cc:407] Connection header received
0618 23:31:54.378278 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:54.378279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:54.378293 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:54.378386 (+ 93us) server_negotiation.cc:226] Negotiated authn=JWT
0618 23:31:54.378602 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:54.379195 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:54.380418 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:54.380628 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:54.380697 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:54.380940 (+ 243us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0618 23:31:54.399167 (+ 18227us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46263/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0618 23:31:54.399466 (+ 299us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41454: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46263/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":74}
W20260618 23:31:54.399669 5601 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41454: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46263/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260618 23:31:54.399996 5611 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:31:54.377744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42203 (local address 127.0.0.1:41454)
0618 23:31:54.378039 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:54.378054 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:54.378160 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:54.378415 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:54.378416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:54.378418 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0618 23:31:54.378521 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:54.378523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:54.379290 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:54.379291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:54.380318 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:54.380320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:54.380715 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:54.380719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:54.380789 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:54.380814 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0618 23:31:54.399364 (+ 18550us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46263/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0618 23:31:54.399591 (+ 227us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:42203: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46263/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":178,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:42203: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46263/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260618 23:31:54.400624 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 5515
2026-06-18T23:31:54Z chronyd exiting
I20260618 23:31:54.437117 392 test_util.cc:175] -----------------------------------------------
I20260618 23:31:54.437213 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825489608381-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (273 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260618 23:31:54.438061 392 mini_oidc.cc:152] Starting JWKS server
I20260618 23:31:54.438865 392 webserver.cc:460] Webserver started at https://127.0.0.1:34973/ using document root <none> and password file <none>
I20260618 23:31:54.438944 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260618 23:31:54.439137 392 webserver.cc:460] Webserver started at http://127.0.0.1:42087/ using document root <none> and password file <none>
2026-06-18T23:31:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:31:54Z Disabled control of system clock
I20260618 23:31:54.448503 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39743
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39743
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34973/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825489608381-392-0/testchainca.cert with env {}
W20260618 23:31:54.534287 5618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:54.534557 5618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:54.534593 5618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:54.535799 5618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:31:54.535888 5618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:54.535910 5618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:54.535929 5618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:31:54.535956 5618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260618 23:31:54.535984 5618 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260618 23:31:54.536007 5618 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34973/jwks.json
I20260618 23:31:54.537595 5618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34973/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825489608381-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:54.537907 5618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:54.538141 5618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:54.544026 5626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:54.544047 5623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:54.544202 5618 server_base.cc:1034] running on GCE node
W20260618 23:31:54.544004 5624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:54.544647 5618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:54.545185 5618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:54.546372 5618 hybrid_clock.cc:648] HybridClock initialized: now 1781825514546346 us; error 55 us; skew 500 ppm
I20260618 23:31:54.548118 5618 webserver.cc:460] Webserver started at http://127.0.98.62:37439/ using document root <none> and password file <none>
I20260618 23:31:54.548432 5618 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:54.548487 5618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:54.548641 5618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:54.550014 5618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "8e070ab3cdc643e5be38497bb0256982"
format_stamp: "Formatted at 2026-06-18 23:31:54 on dist-test-slave-6224"
server_key: "53fe0db4f8e8e74b3c78da2667ddfd06"
server_key_iv: "b20e57c6b784dc8dd7e07b57c7adbd42"
server_key_version: "encryptionkey@0"
I20260618 23:31:54.550446 5618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "8e070ab3cdc643e5be38497bb0256982"
format_stamp: "Formatted at 2026-06-18 23:31:54 on dist-test-slave-6224"
server_key: "53fe0db4f8e8e74b3c78da2667ddfd06"
server_key_iv: "b20e57c6b784dc8dd7e07b57c7adbd42"
server_key_version: "encryptionkey@0"
I20260618 23:31:54.552294 5618 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260618 23:31:54.553364 5632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:54.553728 5618 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:54.553850 5618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "8e070ab3cdc643e5be38497bb0256982"
format_stamp: "Formatted at 2026-06-18 23:31:54 on dist-test-slave-6224"
server_key: "53fe0db4f8e8e74b3c78da2667ddfd06"
server_key_iv: "b20e57c6b784dc8dd7e07b57c7adbd42"
server_key_version: "encryptionkey@0"
I20260618 23:31:54.553966 5618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:54.569084 5618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:54.569566 5618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:54.569757 5618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:54.574827 5618 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39743
I20260618 23:31:54.574864 5684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39743 every 8 connection(s)
I20260618 23:31:54.575275 5618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:31:54.576490 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:54.578959 5685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982: Bootstrap starting.
I20260618 23:31:54.579604 5685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:54.579898 5685 log.cc:824] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:54.580768 5685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982: No bootstrap required, opened a new log
I20260618 23:31:54.582445 5685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e070ab3cdc643e5be38497bb0256982" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39743 } }
I20260618 23:31:54.582615 5685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:54.582653 5685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e070ab3cdc643e5be38497bb0256982, State: Initialized, Role: FOLLOWER
I20260618 23:31:54.582930 5685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e070ab3cdc643e5be38497bb0256982" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39743 } }
I20260618 23:31:54.583148 5685 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:31:54.583209 5685 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:31:54.583267 5685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:54.583884 5685 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e070ab3cdc643e5be38497bb0256982" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39743 } }
I20260618 23:31:54.584007 5685 leader_election.cc:302] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e070ab3cdc643e5be38497bb0256982; no voters:
I20260618 23:31:54.584209 5685 leader_election.cc:288] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:31:54.584319 5688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:54.584499 5685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:31:54.584590 5688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982 [term 1 LEADER]: Becoming Leader. State: Replica: 8e070ab3cdc643e5be38497bb0256982, State: Running, Role: LEADER
I20260618 23:31:54.584769 5688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e070ab3cdc643e5be38497bb0256982" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39743 } }
I20260618 23:31:54.584936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 5618
I20260618 23:31:54.585071 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:31:54.585306 392 external_mini_cluster.cc:1428] Setting key 79d4279ed2c2cd611652f00c4df7d72c
I20260618 23:31:54.585441 5690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e070ab3cdc643e5be38497bb0256982. Latest consensus state: current_term: 1 leader_uuid: "8e070ab3cdc643e5be38497bb0256982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e070ab3cdc643e5be38497bb0256982" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39743 } } }
I20260618 23:31:54.585536 5690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982 [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:54.585737 5689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e070ab3cdc643e5be38497bb0256982" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e070ab3cdc643e5be38497bb0256982" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39743 } } }
I20260618 23:31:54.585810 5689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982 [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:54.585950 5698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:31:54.586975 5698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:31:54.588919 5698 catalog_manager.cc:1347] Generated new cluster ID: b8ff8349f56c4f60a63cf8554725cc1b
I20260618 23:31:54.588982 5698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:31:54.591429 5704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:54.585992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34606 (local address 127.0.98.62:39743)
0618 23:31:54.586574 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:54.586582 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:54.586607 (+ 25us) server_negotiation.cc:407] Connection header received
0618 23:31:54.586705 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:54.586710 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:54.586767 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:54.586922 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:54.587123 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:54.588416 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:54.589366 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:54.589855 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:54.590002 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:54.590723 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:54.590727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:54.590728 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:31:54.590916 (+ 188us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:54.590949 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:31:54.590961 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:54.591038 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:54.591111 (+ 73us) server_negotiation.cc:299] Negotiation successful
0618 23:31:54.591195 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":64,"threads_started":1}
I20260618 23:31:54.592506 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:31:54.608355 5698 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:31:54.608954 5698 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:31:54.615655 5698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8e070ab3cdc643e5be38497bb0256982: Generated new TSK 0
I20260618 23:31:54.615901 5698 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260618 23:31:54.639339 5715 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260618 23:31:54.639473 5715 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260618 23:31:54.639889 5704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:54.624735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34614 (local address 127.0.98.62:39743)
0618 23:31:54.624792 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:54.624794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:54.625019 (+ 225us) server_negotiation.cc:407] Connection header received
0618 23:31:54.625081 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:54.625082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:54.625102 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:54.625229 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0618 23:31:54.625397 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:54.625963 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:54.626756 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:54.626978 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:54.627070 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:54.627397 (+ 327us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0618 23:31:54.639688 (+ 12291us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34973/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0618 23:31:54.639835 (+ 147us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34614: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34973/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":27}
W20260618 23:31:54.639976 5704 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34614: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34973/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260618 23:31:54.640203 5714 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:31:54.624647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39743 (local address 127.0.0.1:34614)
0618 23:31:54.624888 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:54.624901 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:54.625002 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:54.625234 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:54.625235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:54.625237 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0618 23:31:54.625316 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:54.625318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:54.626051 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:54.626052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:54.626663 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:54.626665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:54.627105 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:54.627110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:54.627218 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:54.627228 (+ 10us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0618 23:31:54.639845 (+ 12617us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34973/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0618 23:31:54.640061 (+ 216us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39743: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34973/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":119,"threads_started":1}
I20260618 23:31:54.640602 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 5618
2026-06-18T23:31:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (241 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260618 23:31:54.679210 392 mini_oidc.cc:152] Starting JWKS server
I20260618 23:31:54.680058 392 webserver.cc:460] Webserver started at https://127.0.0.1:41167/ using document root <none> and password file <none>
I20260618 23:31:54.680132 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260618 23:31:54.680290 392 webserver.cc:460] Webserver started at http://127.0.0.1:37567/ using document root <none> and password file <none>
2026-06-18T23:31:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:31:54Z Disabled control of system clock
I20260618 23:31:54.688905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34791
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34791
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41167/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260618 23:31:54.769913 5721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:54.770126 5721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:54.770148 5721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:54.771377 5721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:31:54.771430 5721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:54.771443 5721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:54.771456 5721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:31:54.771471 5721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260618 23:31:54.771482 5721 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260618 23:31:54.771494 5721 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41167/jwks.json
I20260618 23:31:54.772974 5721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41167/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:54.773250 5721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:54.773564 5721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:54.779017 5726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:54.779023 5729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:54.779033 5727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:54.779206 5721 server_base.cc:1034] running on GCE node
I20260618 23:31:54.779714 5721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:54.780049 5721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:54.781208 5721 hybrid_clock.cc:648] HybridClock initialized: now 1781825514781187 us; error 42 us; skew 500 ppm
I20260618 23:31:54.782619 5721 webserver.cc:460] Webserver started at http://127.0.98.62:33251/ using document root <none> and password file <none>
I20260618 23:31:54.782935 5721 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:54.782992 5721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:54.783135 5721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:54.784363 5721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "ef7477b9269d40239b68f3760cad9756"
format_stamp: "Formatted at 2026-06-18 23:31:54 on dist-test-slave-6224"
server_key: "8756f73c5fe357d89e7c33d9dee7753a"
server_key_iv: "8e98fea00873698988a4d52bac471841"
server_key_version: "encryptionkey@0"
I20260618 23:31:54.784726 5721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "ef7477b9269d40239b68f3760cad9756"
format_stamp: "Formatted at 2026-06-18 23:31:54 on dist-test-slave-6224"
server_key: "8756f73c5fe357d89e7c33d9dee7753a"
server_key_iv: "8e98fea00873698988a4d52bac471841"
server_key_version: "encryptionkey@0"
I20260618 23:31:54.786162 5721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:31:54.787118 5735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:54.787403 5721 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:54.787525 5721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "ef7477b9269d40239b68f3760cad9756"
format_stamp: "Formatted at 2026-06-18 23:31:54 on dist-test-slave-6224"
server_key: "8756f73c5fe357d89e7c33d9dee7753a"
server_key_iv: "8e98fea00873698988a4d52bac471841"
server_key_version: "encryptionkey@0"
I20260618 23:31:54.787657 5721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:54.828058 5721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:54.828421 5721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:54.828593 5721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:54.833112 5721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34791
I20260618 23:31:54.833190 5787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34791 every 8 connection(s)
I20260618 23:31:54.833570 5721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:31:54.834348 5788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:54.834854 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 5721
I20260618 23:31:54.834995 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:31:54.835249 392 external_mini_cluster.cc:1428] Setting key ad7cdd1675c97df2b45619f3f4cd5f10
I20260618 23:31:54.838738 5788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756: Bootstrap starting.
I20260618 23:31:54.839618 5788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:54.839980 5788 log.cc:824] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:54.841182 5788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756: No bootstrap required, opened a new log
I20260618 23:31:54.841795 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:54.835887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48482 (local address 127.0.98.62:34791)
0618 23:31:54.836105 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:54.836112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:54.836140 (+ 28us) server_negotiation.cc:407] Connection header received
0618 23:31:54.836745 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:54.836751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:54.836830 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:54.836955 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:54.837270 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:54.838190 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:54.839397 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:54.839970 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:54.840181 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:54.841050 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:54.841055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:54.841055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:31:54.841259 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:54.841314 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:31:54.841326 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:54.841408 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:54.841492 (+ 84us) server_negotiation.cc:299] Negotiation successful
0618 23:31:54.841584 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
I20260618 23:31:54.842993 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:31:54.844041 5788 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7477b9269d40239b68f3760cad9756" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34791 } }
I20260618 23:31:54.844269 5788 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:54.844319 5788 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef7477b9269d40239b68f3760cad9756, State: Initialized, Role: FOLLOWER
I20260618 23:31:54.844458 5788 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7477b9269d40239b68f3760cad9756" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34791 } }
I20260618 23:31:54.844542 5788 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:31:54.844568 5788 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:31:54.844599 5788 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:54.845360 5788 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7477b9269d40239b68f3760cad9756" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34791 } }
I20260618 23:31:54.845495 5788 leader_election.cc:302] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef7477b9269d40239b68f3760cad9756; no voters:
I20260618 23:31:54.845736 5788 leader_election.cc:288] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:31:54.845794 5793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:54.845950 5793 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756 [term 1 LEADER]: Becoming Leader. State: Replica: ef7477b9269d40239b68f3760cad9756, State: Running, Role: LEADER
I20260618 23:31:54.846033 5793 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7477b9269d40239b68f3760cad9756" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34791 } }
I20260618 23:31:54.846073 5788 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:31:54.846542 5794 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef7477b9269d40239b68f3760cad9756" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7477b9269d40239b68f3760cad9756" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34791 } } }
I20260618 23:31:54.846536 5795 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef7477b9269d40239b68f3760cad9756. Latest consensus state: current_term: 1 leader_uuid: "ef7477b9269d40239b68f3760cad9756" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef7477b9269d40239b68f3760cad9756" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34791 } } }
I20260618 23:31:54.846719 5794 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756 [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:54.846746 5795 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756 [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:54.847705 5801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:31:54.848284 5801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:31:54.850243 5801 catalog_manager.cc:1347] Generated new cluster ID: 97d5e4510163418ca2d0edbce6760df5
I20260618 23:31:54.850311 5801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:31:54.865582 5801 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:31:54.866223 5801 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:31:54.868999 5801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef7477b9269d40239b68f3760cad9756: Generated new TSK 0
I20260618 23:31:54.869184 5801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:31:54.880836 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:54.876359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48488 (local address 127.0.98.62:34791)
0618 23:31:54.876449 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:54.876452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:54.876539 (+ 87us) server_negotiation.cc:407] Connection header received
0618 23:31:54.876635 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:54.876637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:54.876653 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:54.876737 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0618 23:31:54.876964 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:54.877834 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:54.878748 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:54.879079 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:54.879156 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:54.879378 (+ 222us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0618 23:31:54.880575 (+ 1197us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41167/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0618 23:31:54.880731 (+ 156us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48488: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41167/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":58}
W20260618 23:31:54.880919 5791 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48488: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41167/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260618 23:31:54.882797 5817 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:31:54.876227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34791 (local address 127.0.0.1:48488)
0618 23:31:54.876429 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:54.876445 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:54.876537 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:54.876749 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:54.876750 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:54.876752 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0618 23:31:54.876839 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:54.876841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:54.877953 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:54.877955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:54.878646 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:54.878649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:54.879174 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:54.879182 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:54.879242 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:54.879249 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0618 23:31:54.880683 (+ 1434us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41167/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0618 23:31:54.882684 (+ 2001us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34791: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41167/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":1766,"thread_start_us":101,"threads_started":1}
I20260618 23:31:54.883359 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 5721
E20260618 23:31:54.891875 5818 webserver.cc:533] Webserver: sslize failed:
E20260618 23:31:54.891968 5818 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-18T23:31:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (242 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-18T23:31:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:31:54Z Disabled control of system clock
I20260618 23:31:54.930740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39425
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39425
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825489608381-392-0/testchainca.cert with env {}
W20260618 23:31:55.021343 5822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:55.021670 5822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:55.021709 5822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:55.023156 5822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:31:55.023208 5822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:55.023231 5822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:55.023267 5822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:31:55.023286 5822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260618 23:31:55.023304 5822 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260618 23:31:55.023353 5822 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260618 23:31:55.025650 5822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825489608381-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:55.025945 5822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:55.026243 5822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:55.032115 5827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:55.032455 5828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:55.032136 5830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:55.037007 5822 server_base.cc:1034] running on GCE node
I20260618 23:31:55.037379 5822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:55.037801 5822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:55.038995 5822 hybrid_clock.cc:648] HybridClock initialized: now 1781825515038962 us; error 55 us; skew 500 ppm
I20260618 23:31:55.040684 5822 webserver.cc:460] Webserver started at http://127.0.98.62:36339/ using document root <none> and password file <none>
I20260618 23:31:55.040952 5822 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:55.041021 5822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:55.041175 5822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:55.042382 5822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "dcb0c9d554c0454a8abe247362653f44"
format_stamp: "Formatted at 2026-06-18 23:31:55 on dist-test-slave-6224"
server_key: "3d467594d57c70b23374dcb3cb47c77e"
server_key_iv: "7117efb6d73bb69fcb130ef4e34e41c3"
server_key_version: "encryptionkey@0"
I20260618 23:31:55.042795 5822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "dcb0c9d554c0454a8abe247362653f44"
format_stamp: "Formatted at 2026-06-18 23:31:55 on dist-test-slave-6224"
server_key: "3d467594d57c70b23374dcb3cb47c77e"
server_key_iv: "7117efb6d73bb69fcb130ef4e34e41c3"
server_key_version: "encryptionkey@0"
I20260618 23:31:55.044492 5822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:31:55.045383 5836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:55.045636 5822 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:55.045751 5822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "dcb0c9d554c0454a8abe247362653f44"
format_stamp: "Formatted at 2026-06-18 23:31:55 on dist-test-slave-6224"
server_key: "3d467594d57c70b23374dcb3cb47c77e"
server_key_iv: "7117efb6d73bb69fcb130ef4e34e41c3"
server_key_version: "encryptionkey@0"
I20260618 23:31:55.045845 5822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:55.056258 5822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:55.056610 5822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:55.056773 5822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:55.061712 5822 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39425
I20260618 23:31:55.061765 5888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39425 every 8 connection(s)
I20260618 23:31:55.062181 5822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:31:55.063500 5889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:55.066007 5889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44: Bootstrap starting.
I20260618 23:31:55.066272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 5822
I20260618 23:31:55.066406 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:31:55.066624 392 external_mini_cluster.cc:1428] Setting key 176c5fbeff565a98195ef699e16ded54
I20260618 23:31:55.066776 5889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:55.067099 5889 log.cc:824] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:55.068028 5889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44: No bootstrap required, opened a new log
I20260618 23:31:55.070528 5889 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb0c9d554c0454a8abe247362653f44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } }
I20260618 23:31:55.070667 5889 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:55.070689 5889 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcb0c9d554c0454a8abe247362653f44, State: Initialized, Role: FOLLOWER
I20260618 23:31:55.070819 5889 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb0c9d554c0454a8abe247362653f44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } }
I20260618 23:31:55.070902 5889 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:31:55.070925 5889 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:31:55.070995 5889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:55.071679 5889 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb0c9d554c0454a8abe247362653f44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } }
I20260618 23:31:55.071786 5889 leader_election.cc:302] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcb0c9d554c0454a8abe247362653f44; no voters:
I20260618 23:31:55.071956 5889 leader_election.cc:288] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:31:55.072014 5894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:55.072209 5894 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44 [term 1 LEADER]: Becoming Leader. State: Replica: dcb0c9d554c0454a8abe247362653f44, State: Running, Role: LEADER
I20260618 23:31:55.072270 5889 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:31:55.072338 5894 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb0c9d554c0454a8abe247362653f44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } }
I20260618 23:31:55.072763 5894 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcb0c9d554c0454a8abe247362653f44. Latest consensus state: current_term: 1 leader_uuid: "dcb0c9d554c0454a8abe247362653f44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb0c9d554c0454a8abe247362653f44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } } }
I20260618 23:31:55.072847 5894 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44 [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:55.073115 5901 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:31:55.073731 5901 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:31:55.074126 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:55.067263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58582 (local address 127.0.98.62:39425)
0618 23:31:55.067616 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:55.067625 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:55.067650 (+ 25us) server_negotiation.cc:407] Connection header received
0618 23:31:55.068320 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:55.068329 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:55.068403 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:55.068567 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:55.068840 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:55.070304 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:55.071412 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:55.072125 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:55.072655 (+ 530us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:55.073180 (+ 525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:55.073185 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:55.073185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:31:55.073424 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:55.073470 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:31:55.073487 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:55.073553 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:55.073789 (+ 236us) server_negotiation.cc:299] Negotiation successful
0618 23:31:55.073889 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":104,"threads_started":1}
I20260618 23:31:55.073092 5895 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcb0c9d554c0454a8abe247362653f44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcb0c9d554c0454a8abe247362653f44" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } } }
I20260618 23:31:55.074699 5895 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44 [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:55.075203 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:31:55.075904 5901 catalog_manager.cc:1347] Generated new cluster ID: 47dd0cfd93394ecb8279bab1ecf9ea28
I20260618 23:31:55.075965 5901 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:31:55.089226 5901 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:31:55.089828 5901 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:31:55.094398 5901 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcb0c9d554c0454a8abe247362653f44: Generated new TSK 0
I20260618 23:31:55.094583 5901 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:31:55.523608 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:55.102436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58590 (local address 127.0.98.62:39425)
0618 23:31:55.102515 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:55.102517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:55.102706 (+ 189us) server_negotiation.cc:407] Connection header received
0618 23:31:55.102823 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:55.102824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:55.102835 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:55.102940 (+ 105us) server_negotiation.cc:226] Negotiated authn=JWT
0618 23:31:55.103091 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:55.103691 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:55.105098 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:55.105297 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:55.105361 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:55.105584 (+ 223us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0618 23:31:55.523297 (+417713us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0618 23:31:55.523484 (+ 187us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58590: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":50}
W20260618 23:31:55.523701 5893 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58590: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260618 23:31:55.523949 5918 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:31:55.102355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39425 (local address 127.0.0.1:58590)
0618 23:31:55.102591 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:55.102606 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:55.102723 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:55.102935 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:55.102936 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:55.102939 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0618 23:31:55.103020 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:55.103023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:55.103803 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:55.103804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:55.104991 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:55.104995 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:55.105394 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:55.105402 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:55.105477 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:55.105485 (+ 8us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0618 23:31:55.523511 (+418026us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0618 23:31:55.523812 (+ 301us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39425: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":146,"threads_started":1}
I20260618 23:31:55.524493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 5822
2026-06-18T23:31:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (621 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (97 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (97 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:31:55 dist-test-slave-6224 krb5kdc[5930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:31:55 dist-test-slave-6224 krb5kdc[5930](info): set up 2 sockets
Jun 18 23:31:55 dist-test-slave-6224 krb5kdc[5930](info): commencing operation
krb5kdc: starting...
W20260618 23:31:56.767864 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:31:56 dist-test-slave-6224 krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:31:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:31:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:31:56.898293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43443
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43443
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:56.979661 5943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:56.979933 5943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:56.979969 5943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:56.981153 5943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:31:56.981230 5943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:56.981253 5943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:56.981274 5943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:31:56.981328 5943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:31:56.982877 5943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:56.983173 5943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:56.983395 5943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:56.989060 5951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:56.989208 5943 server_base.cc:1034] running on GCE node
W20260618 23:31:56.989109 5948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:56.989109 5949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:56.989662 5943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:56.990074 5943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:56.991262 5943 hybrid_clock.cc:648] HybridClock initialized: now 1781825516991236 us; error 53 us; skew 500 ppm
Jun 18 23:31:56 dist-test-slave-6224 krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825516, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:56.994122 5943 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:31:56.994665 5943 webserver.cc:460] Webserver started at http://127.0.98.62:40985/ using document root <none> and password file <none>
I20260618 23:31:56.994927 5943 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:56.994998 5943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:56.995152 5943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:56.996466 5943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "d73bfec4aa5c4035bb75d74987c88531"
format_stamp: "Formatted at 2026-06-18 23:31:56 on dist-test-slave-6224"
server_key: "bc9a6fad2548e41c1350896e6a6dff27"
server_key_iv: "193c414bc6d95819dd324c4a7c8d9bdc"
server_key_version: "encryptionkey@0"
I20260618 23:31:56.996891 5943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "d73bfec4aa5c4035bb75d74987c88531"
format_stamp: "Formatted at 2026-06-18 23:31:56 on dist-test-slave-6224"
server_key: "bc9a6fad2548e41c1350896e6a6dff27"
server_key_iv: "193c414bc6d95819dd324c4a7c8d9bdc"
server_key_version: "encryptionkey@0"
I20260618 23:31:56.998440 5943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260618 23:31:56.999333 5958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:56.999635 5943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:56.999750 5943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "d73bfec4aa5c4035bb75d74987c88531"
format_stamp: "Formatted at 2026-06-18 23:31:56 on dist-test-slave-6224"
server_key: "bc9a6fad2548e41c1350896e6a6dff27"
server_key_iv: "193c414bc6d95819dd324c4a7c8d9bdc"
server_key_version: "encryptionkey@0"
I20260618 23:31:56.999835 5943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:57.018047 5943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:57.020547 5943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:57.020740 5943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:57.025795 5943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43443
I20260618 23:31:57.025859 6010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43443 every 8 connection(s)
I20260618 23:31:57.026455 5943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:31:57.027343 6011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:57.029840 6011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531: Bootstrap starting.
I20260618 23:31:57.030426 6011 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:57.030687 6011 log.cc:824] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:57.031385 6011 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531: No bootstrap required, opened a new log
I20260618 23:31:57.033002 6011 raft_consensus.cc:348] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bfec4aa5c4035bb75d74987c88531" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43443 } }
I20260618 23:31:57.033152 6011 raft_consensus.cc:374] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:57.033175 6011 raft_consensus.cc:729] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d73bfec4aa5c4035bb75d74987c88531, State: Initialized, Role: FOLLOWER
I20260618 23:31:57.033321 6011 consensus_queue.cc:260] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bfec4aa5c4035bb75d74987c88531" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43443 } }
I20260618 23:31:57.033409 6011 raft_consensus.cc:388] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:31:57.033432 6011 raft_consensus.cc:482] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:31:57.033460 6011 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:57.034334 6011 raft_consensus.cc:504] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bfec4aa5c4035bb75d74987c88531" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43443 } }
I20260618 23:31:57.034421 6011 leader_election.cc:302] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d73bfec4aa5c4035bb75d74987c88531; no voters:
I20260618 23:31:57.034606 6011 leader_election.cc:288] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:31:57.034729 6014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:57.034778 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 5943
I20260618 23:31:57.034895 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:31:57.034922 6011 sys_catalog.cc:564] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:31:57.034988 6014 raft_consensus.cc:686] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531 [term 1 LEADER]: Becoming Leader. State: Replica: d73bfec4aa5c4035bb75d74987c88531, State: Running, Role: LEADER
I20260618 23:31:57.035167 392 external_mini_cluster.cc:1428] Setting key 96b045870f62ce36397aa3444047d50d
I20260618 23:31:57.035167 6014 consensus_queue.cc:237] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bfec4aa5c4035bb75d74987c88531" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43443 } }
I20260618 23:31:57.035688 6015 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d73bfec4aa5c4035bb75d74987c88531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bfec4aa5c4035bb75d74987c88531" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43443 } } }
I20260618 23:31:57.035802 6015 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531 [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:57.035966 6016 sys_catalog.cc:455] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d73bfec4aa5c4035bb75d74987c88531. Latest consensus state: current_term: 1 leader_uuid: "d73bfec4aa5c4035bb75d74987c88531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d73bfec4aa5c4035bb75d74987c88531" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43443 } } }
I20260618 23:31:57.036055 6016 sys_catalog.cc:458] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531 [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:57.036726 6025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:31:57.037998 6025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:31:57.040961 6025 catalog_manager.cc:1347] Generated new cluster ID: 05bec20949984ea3b766cd06f243d4fd
I20260618 23:31:57.041047 6025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 18 23:31:57 dist-test-slave-6224 krb5kdc[5930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:57.050652 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:57.035842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.0.98.62:43443)
0618 23:31:57.036218 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:57.036228 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:57.036374 (+ 146us) server_negotiation.cc:407] Connection header received
0618 23:31:57.036655 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:57.036665 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:57.036741 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:57.036901 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:57.038014 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:57.039349 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:57.040533 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:57.041217 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:57.041497 (+ 280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:57.045675 (+ 4178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:57.045680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:57.045681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:57.045697 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:57.048242 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:57.049024 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:57.049028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:57.049029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:57.049181 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:57.049446 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:57.049446 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:57.049447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:57.049878 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:57.050083 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:57.050180 (+ 97us) server_negotiation.cc:299] Negotiation successful
0618 23:31:57.050357 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":86,"threads_started":1}
I20260618 23:31:57.051051 6025 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:31:57.051836 6025 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:31:57.057662 6025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d73bfec4aa5c4035bb75d74987c88531: Generated new TSK 0
I20260618 23:31:57.057906 6025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260618 23:31:57.094774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43443
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:57.178560 6037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:57.178822 6037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:57.178858 6037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:57.180055 6037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:57.180130 6037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:57.180176 6037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:31:57.181703 6037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:57.182003 6037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:57.182245 6037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:57.188622 6043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:57.188745 6045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:57.188774 6042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:57.189078 6037 server_base.cc:1034] running on GCE node
I20260618 23:31:57.189285 6037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:57.189596 6037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:57.190765 6037 hybrid_clock.cc:648] HybridClock initialized: now 1781825517190743 us; error 39 us; skew 500 ppm
Jun 18 23:31:57 dist-test-slave-6224 krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:57.193398 6037 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:31:57.193914 6037 webserver.cc:460] Webserver started at http://127.0.98.1:35973/ using document root <none> and password file <none>
I20260618 23:31:57.194160 6037 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:57.194211 6037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:57.194402 6037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:57.195643 6037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "f0f219001593487ca6c1f61cc9b69598"
format_stamp: "Formatted at 2026-06-18 23:31:57 on dist-test-slave-6224"
server_key: "4167028bf38696a817ebde496d58c3dc"
server_key_iv: "b5c791fa9ad199ed5cac8f5e0c1d8a21"
server_key_version: "encryptionkey@0"
I20260618 23:31:57.196024 6037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f0f219001593487ca6c1f61cc9b69598"
format_stamp: "Formatted at 2026-06-18 23:31:57 on dist-test-slave-6224"
server_key: "4167028bf38696a817ebde496d58c3dc"
server_key_iv: "b5c791fa9ad199ed5cac8f5e0c1d8a21"
server_key_version: "encryptionkey@0"
I20260618 23:31:57.197546 6037 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260618 23:31:57.198536 6052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:57.198899 6037 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:31:57.199000 6037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "f0f219001593487ca6c1f61cc9b69598"
format_stamp: "Formatted at 2026-06-18 23:31:57 on dist-test-slave-6224"
server_key: "4167028bf38696a817ebde496d58c3dc"
server_key_iv: "b5c791fa9ad199ed5cac8f5e0c1d8a21"
server_key_version: "encryptionkey@0"
I20260618 23:31:57.199110 6037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:57.219156 6037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:57.221694 6037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:57.221825 6037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:57.222155 6037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:57.222502 6037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:57.222544 6037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:57.222568 6037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:57.222580 6037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:57.229470 6037 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33093
I20260618 23:31:57.229669 6165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33093 every 8 connection(s)
I20260618 23:31:57.229923 6037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:31:57.231222 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 6037
I20260618 23:31:57.231369 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:31:57.231632 392 external_mini_cluster.cc:1428] Setting key 6b4d28a1d9acbc823dc1f4634772e9f6
Jun 18 23:31:57 dist-test-slave-6224 krb5kdc[5930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260618 23:31:57.242273 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:57.230962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45323 (local address 127.0.98.62:43443)
0618 23:31:57.231049 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:57.231052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:57.231404 (+ 352us) server_negotiation.cc:407] Connection header received
0618 23:31:57.232718 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:57.232719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:57.232731 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:57.232770 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:57.233796 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:57.234575 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:57.235808 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:57.236178 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:57.236264 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:57.238762 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:57.238765 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:57.238766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:57.238782 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:57.240652 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:57.241139 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:57.241140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:57.241141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:57.241207 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:57.241463 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:57.241464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:57.241464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:57.241637 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:57.241739 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:57.242027 (+ 288us) server_negotiation.cc:299] Negotiation successful
0618 23:31:57.242131 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 23:31:57.242435 6168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:57.230799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43443 (local address 127.0.98.1:45323)
0618 23:31:57.231142 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:57.231233 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:57.232585 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:57.232873 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:57.232878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:57.233353 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:57.233682 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:57.233688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:57.234686 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:57.234687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:57.235679 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:57.235682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:57.236307 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:57.236308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:57.236462 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:57.237111 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:57.237121 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:57.238629 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:57.240775 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:57.240781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:57.240783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:57.241050 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:57.241287 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:57.241288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:57.241288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:57.241387 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:57.241723 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:57.241727 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:57.241851 (+ 124us) client_negotiation.cc:769] Sending connection context
0618 23:31:57.241975 (+ 124us) client_negotiation.cc:240] Negotiation successful
0618 23:31:57.242115 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":94,"threads_started":1}
I20260618 23:31:57.243255 6166 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43443
I20260618 23:31:57.243391 6166 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:57.243652 6166 heartbeater.cc:511] Master 127.0.98.62:43443 requested a full tablet report, sending...
I20260618 23:31:57.244341 5975 ts_manager.cc:194] Registered new tserver with Master: f0f219001593487ca6c1f61cc9b69598 (127.0.98.1:33093)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260618 23:31:57.245035 5975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260618 23:31:57.263610 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43443
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:57.346722 6171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:57.346987 6171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:57.347025 6171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:57.348243 6171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:57.348316 6171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:57.348364 6171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:31:57.349888 6171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:57.350178 6171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:57.350418 6171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:57.355774 6176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:57.355882 6177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:57.355882 6179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:57.356202 6171 server_base.cc:1034] running on GCE node
I20260618 23:31:57.356514 6171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:57.356791 6171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:57.357997 6171 hybrid_clock.cc:648] HybridClock initialized: now 1781825517357982 us; error 46 us; skew 500 ppm
Jun 18 23:31:57 dist-test-slave-6224 krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:57.361239 6171 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:31:57.361799 6171 webserver.cc:460] Webserver started at http://127.0.98.2:35237/ using document root <none> and password file <none>
I20260618 23:31:57.362067 6171 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:57.362138 6171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:57.362309 6171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:57.363615 6171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "92dff0e488be458a94455e2012351b3b"
format_stamp: "Formatted at 2026-06-18 23:31:57 on dist-test-slave-6224"
server_key: "744d443d6019103c436b84fad1eedc9c"
server_key_iv: "6a3b599d09fbb97ea46e658198df9e88"
server_key_version: "encryptionkey@0"
I20260618 23:31:57.364040 6171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "92dff0e488be458a94455e2012351b3b"
format_stamp: "Formatted at 2026-06-18 23:31:57 on dist-test-slave-6224"
server_key: "744d443d6019103c436b84fad1eedc9c"
server_key_iv: "6a3b599d09fbb97ea46e658198df9e88"
server_key_version: "encryptionkey@0"
I20260618 23:31:57.365720 6171 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260618 23:31:57.366672 6186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:57.366932 6171 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:57.367056 6171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "92dff0e488be458a94455e2012351b3b"
format_stamp: "Formatted at 2026-06-18 23:31:57 on dist-test-slave-6224"
server_key: "744d443d6019103c436b84fad1eedc9c"
server_key_iv: "6a3b599d09fbb97ea46e658198df9e88"
server_key_version: "encryptionkey@0"
I20260618 23:31:57.367141 6171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:57.378470 6171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:57.380761 6171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:57.380895 6171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:57.381119 6171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:57.381489 6171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:57.381551 6171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:57.381598 6171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:57.381625 6171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:57.389123 6171 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38839
I20260618 23:31:57.389178 6299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38839 every 8 connection(s)
I20260618 23:31:57.389611 6171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
Jun 18 23:31:57 dist-test-slave-6224 krb5kdc[5930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:57.399466 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 6171
I20260618 23:31:57.399643 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:31:57.399914 392 external_mini_cluster.cc:1428] Setting key 5e676e174a333a166941aed0fbc4f6b6
I20260618 23:31:57.399964 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:57.390528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57249 (local address 127.0.98.62:43443)
0618 23:31:57.390594 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:57.390597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:57.390813 (+ 216us) server_negotiation.cc:407] Connection header received
0618 23:31:57.391957 (+ 1144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:57.391958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:57.391970 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:57.392040 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:57.392710 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:57.393229 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:57.394234 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:57.394425 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:57.394501 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:57.396926 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:57.396928 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:57.396928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:57.396936 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:57.398410 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:57.398881 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:57.398882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:57.398882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:57.398948 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:57.399197 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:57.399197 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:57.399198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:57.399319 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:57.399400 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:57.399775 (+ 375us) server_negotiation.cc:299] Negotiation successful
0618 23:31:57.399864 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260618 23:31:57.400110 6302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:57.390482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43443 (local address 127.0.98.2:57249)
0618 23:31:57.390671 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:57.390713 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:57.391850 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:57.392124 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:57.392127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:57.392404 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:57.392619 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:57.392622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:57.393322 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:57.393323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:57.394135 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:57.394137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:57.394506 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:57.394507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:57.394630 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:57.395201 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:57.395219 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:57.396802 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:57.398511 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:57.398525 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:57.398527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:57.398761 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:57.399015 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:57.399016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:57.399016 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:57.399127 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:57.399388 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:57.399394 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:57.399589 (+ 195us) client_negotiation.cc:769] Sending connection context
0618 23:31:57.399711 (+ 122us) client_negotiation.cc:240] Negotiation successful
0618 23:31:57.399876 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":86,"threads_started":1}
I20260618 23:31:57.400838 6300 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43443
I20260618 23:31:57.400959 6300 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:57.401180 6300 heartbeater.cc:511] Master 127.0.98.62:43443 requested a full tablet report, sending...
I20260618 23:31:57.401654 5975 ts_manager.cc:194] Registered new tserver with Master: 92dff0e488be458a94455e2012351b3b (127.0.98.2:38839)
I20260618 23:31:57.402159 5975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57249
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260618 23:31:57.426220 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43443
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:57.508256 6305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:57.508477 6305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:57.508502 6305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:57.509685 6305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:57.509734 6305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:57.509760 6305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:31:57.511273 6305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:57.511509 6305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:57.511792 6305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:57.517328 6311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:57.517398 6310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:57.517340 6313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:57.517756 6305 server_base.cc:1034] running on GCE node
I20260618 23:31:57.518028 6305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:57.518335 6305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:57.519495 6305 hybrid_clock.cc:648] HybridClock initialized: now 1781825517519466 us; error 47 us; skew 500 ppm
Jun 18 23:31:57 dist-test-slave-6224 krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:57.522440 6305 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:31:57.522912 6305 webserver.cc:460] Webserver started at http://127.0.98.3:39923/ using document root <none> and password file <none>
I20260618 23:31:57.523175 6305 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:57.523247 6305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:57.523396 6305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:57.524677 6305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "79f28273b4a24fc8b03c9f61ee0e3a62"
format_stamp: "Formatted at 2026-06-18 23:31:57 on dist-test-slave-6224"
server_key: "2c7a59464226caa6e5d472db00cbec78"
server_key_iv: "362f9a8d953968f8fcaf4302e5ed442c"
server_key_version: "encryptionkey@0"
I20260618 23:31:57.525081 6305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "79f28273b4a24fc8b03c9f61ee0e3a62"
format_stamp: "Formatted at 2026-06-18 23:31:57 on dist-test-slave-6224"
server_key: "2c7a59464226caa6e5d472db00cbec78"
server_key_iv: "362f9a8d953968f8fcaf4302e5ed442c"
server_key_version: "encryptionkey@0"
I20260618 23:31:57.526681 6305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260618 23:31:57.527689 6320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:57.527978 6305 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260618 23:31:57.528091 6305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "79f28273b4a24fc8b03c9f61ee0e3a62"
format_stamp: "Formatted at 2026-06-18 23:31:57 on dist-test-slave-6224"
server_key: "2c7a59464226caa6e5d472db00cbec78"
server_key_iv: "362f9a8d953968f8fcaf4302e5ed442c"
server_key_version: "encryptionkey@0"
I20260618 23:31:57.528191 6305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:57.541831 6305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:57.544443 6305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:57.544626 6305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:57.544900 6305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:57.545256 6305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:57.545312 6305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:57.545362 6305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:57.545388 6305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:57.553524 6305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42119
I20260618 23:31:57.553580 6433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42119 every 8 connection(s)
I20260618 23:31:57.553983 6305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:31:57.562395 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 6305
I20260618 23:31:57.562507 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:31:57.562711 392 external_mini_cluster.cc:1428] Setting key 0650736c680ce08ccffe58f12ae1c652
Jun 18 23:31:57 dist-test-slave-6224 krb5kdc[5930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825517, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:31:57.567592 6024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:57.554923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57047 (local address 127.0.98.62:43443)
0618 23:31:57.554986 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:57.555005 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:57.555245 (+ 240us) server_negotiation.cc:407] Connection header received
0618 23:31:57.556482 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:57.556483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:57.556499 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:57.556549 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:57.557431 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:57.558012 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:57.559793 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:57.560205 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:57.560327 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:57.563758 (+ 3431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:57.563760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:57.563760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:57.563772 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:57.565713 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:57.566354 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:57.566356 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:57.566356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:57.566430 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:57.566727 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:57.566728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:57.566729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:57.566913 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:57.567003 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:57.567307 (+ 304us) server_negotiation.cc:299] Negotiation successful
0618 23:31:57.567415 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:31:57.567627 6436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:57.554882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43443 (local address 127.0.98.3:57047)
0618 23:31:57.555076 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:57.555128 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:57.556341 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:57.556636 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:57.556640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:57.557056 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:57.557329 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:57.557334 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:57.558114 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:57.558115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:57.559653 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:57.559656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:57.560355 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:57.560356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:57.560505 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:57.561199 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:57.561210 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:57.563611 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:57.565853 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:57.565863 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:57.565866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:57.566214 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:57.566516 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:57.566517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:57.566517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:57.566631 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:57.566993 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:57.566998 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:57.567141 (+ 143us) client_negotiation.cc:769] Sending connection context
0618 23:31:57.567230 (+ 89us) client_negotiation.cc:240] Negotiation successful
0618 23:31:57.567386 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":94,"threads_started":1}
I20260618 23:31:57.568252 6434 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43443
I20260618 23:31:57.568418 6434 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:57.568713 6434 heartbeater.cc:511] Master 127.0.98.62:43443 requested a full tablet report, sending...
I20260618 23:31:57.569248 5975 ts_manager.cc:194] Registered new tserver with Master: 79f28273b4a24fc8b03c9f61ee0e3a62 (127.0.98.3:42119)
I20260618 23:31:57.569962 5975 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57047
I20260618 23:31:57.575023 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/test-admin.keytab.
Jun 18 23:31:57 dist-test-slave-6224 krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:57.592730 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260618 23:31:57.594648 6441 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:31:57.593749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46002 (local address 127.0.98.1:33093)
0618 23:31:57.593987 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:57.593993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:57.594003 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:31:57.594037 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:57.594039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:57.594052 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:57.594137 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:57.594552 (+ 415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46002: BlockingRecv error: recv got EOF from 127.0.0.1:46002 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":92,"threads_started":1}
W20260618 23:31:57.594888 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/krb5cc' of type FILE
Jun 18 23:31:57 dist-test-slave-6224 krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:57.596201 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 18 23:31:57 dist-test-slave-6224 krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:57.597124 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 18 23:31:57 dist-test-slave-6224 krb5kdc[5930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260618 23:31:57.606371 6441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:57.597851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46008 (local address 127.0.98.1:33093)
0618 23:31:57.597947 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:57.597949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:57.597959 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:31:57.598016 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:57.598016 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:57.598026 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:57.598083 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:57.598594 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:57.599372 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:57.600271 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:57.600615 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:57.600682 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:57.603288 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:57.603294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:57.603295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:57.603302 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:57.605134 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:57.605556 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:57.605559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:57.605559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:57.605624 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:57.605828 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:57.605828 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:57.605828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:57.605972 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:57.606097 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:57.606203 (+ 106us) server_negotiation.cc:299] Negotiation successful
0618 23:31:57.606279 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260618 23:31:57.609009 6441 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:31:57.608196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46024 (local address 127.0.98.1:33093)
0618 23:31:57.608271 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:57.608273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:57.608398 (+ 125us) server_negotiation.cc:407] Connection header received
0618 23:31:57.608453 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:57.608454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:57.608465 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:57.608520 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:57.608931 (+ 411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46024: BlockingRecv error: recv got EOF from 127.0.0.1:46024 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260618 23:31:57.609110 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/krb5cc)
Jun 18 23:31:57 dist-test-slave-6224 krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:57.610162 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 18 23:31:57 dist-test-slave-6224 krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:57.611001 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 18 23:31:57 dist-test-slave-6224 krb5kdc[5930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260618 23:31:57.619192 6441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:57.611639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46040 (local address 127.0.98.1:33093)
0618 23:31:57.611717 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:57.611719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:57.611890 (+ 171us) server_negotiation.cc:407] Connection header received
0618 23:31:57.611984 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:57.611985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:57.611997 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:57.612054 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:57.612423 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:57.612960 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:57.613779 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:57.614021 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:57.614092 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:57.616514 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:57.616515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:57.616515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:57.616523 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:57.617926 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:57.618297 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:57.618298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:57.618298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:57.618355 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:57.618543 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:57.618544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:57.618544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:57.618658 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:57.618747 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:57.618951 (+ 204us) server_negotiation.cc:299] Negotiation successful
0618 23:31:57.619033 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260618 23:31:57.621186 6441 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:31:57.620547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46044 (local address 127.0.98.1:33093)
0618 23:31:57.620594 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:57.620596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:57.620685 (+ 89us) server_negotiation.cc:407] Connection header received
0618 23:31:57.620757 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:57.620763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:57.620772 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:57.620817 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:57.621119 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46044: BlockingRecv error: recv got EOF from 127.0.0.1:46044 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260618 23:31:57.621321 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825489608381-392-0/krb5kdc/krb5cc' of type FILE
Jun 18 23:31:57 dist-test-slave-6224 krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:57.622440 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 18 23:31:57 dist-test-slave-6224 krb5kdc[5930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:57.623324 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 18 23:31:57 dist-test-slave-6224 krb5kdc[5930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825517, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260618 23:31:57.631441 6441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:57.623985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46046 (local address 127.0.98.1:33093)
0618 23:31:57.624092 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:57.624095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:57.624197 (+ 102us) server_negotiation.cc:407] Connection header received
0618 23:31:57.624302 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:57.624303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:57.624315 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:57.624375 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:57.624811 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:57.625355 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:57.626097 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:57.626316 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:57.626400 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:57.628789 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:57.628791 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:57.628791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:57.628800 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:57.630140 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:57.630530 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:57.630532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:57.630532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:57.630624 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:57.630826 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:57.630828 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:57.630828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:57.630985 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:57.631092 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:57.631232 (+ 140us) server_negotiation.cc:299] Negotiation successful
0618 23:31:57.631329 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260618 23:31:57.632336 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 6037
I20260618 23:31:57.638391 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 6171
I20260618 23:31:57.643360 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 6305
I20260618 23:31:57.648782 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 5943
2026-06-18T23:31:57Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1931 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:31:57 dist-test-slave-6224 krb5kdc[6454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:31:57 dist-test-slave-6224 krb5kdc[6454](info): set up 2 sockets
Jun 18 23:31:57 dist-test-slave-6224 krb5kdc[6454](info): commencing operation
krb5kdc: starting...
W20260618 23:31:58.688424 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:31:58 dist-test-slave-6224 krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:31:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:31:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260618 23:31:58.814896 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43955
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:58.903221 6467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:58.903482 6467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:58.903506 6467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:58.905018 6467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:31:58.905092 6467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:58.905108 6467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:58.905122 6467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:31:58.905135 6467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:31:58.908079 6467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:58.908344 6467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:58.908555 6467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:58.914609 6475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:58.914680 6467 server_base.cc:1034] running on GCE node
W20260618 23:31:58.914654 6473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:58.914757 6472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:58.915172 6467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:58.915586 6467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:58.916774 6467 hybrid_clock.cc:648] HybridClock initialized: now 1781825518916762 us; error 35 us; skew 500 ppm
Jun 18 23:31:58 dist-test-slave-6224 krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825518, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:58.924417 6467 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260618 23:31:58.924998 6467 webserver.cc:460] Webserver started at http://127.0.98.62:35017/ using document root <none> and password file <none>
I20260618 23:31:58.925252 6467 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:58.925324 6467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:58.925488 6467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:58.926633 6467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "2270c132876245d0b99153c4cc9f43db"
format_stamp: "Formatted at 2026-06-18 23:31:58 on dist-test-slave-6224"
server_key: "502461623d7c5713d8f2f762bc283649"
server_key_iv: "9348ca0cdcc5a1ac138c31801b4a871a"
server_key_version: "encryptionkey@0"
I20260618 23:31:58.927021 6467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "2270c132876245d0b99153c4cc9f43db"
format_stamp: "Formatted at 2026-06-18 23:31:58 on dist-test-slave-6224"
server_key: "502461623d7c5713d8f2f762bc283649"
server_key_iv: "9348ca0cdcc5a1ac138c31801b4a871a"
server_key_version: "encryptionkey@0"
I20260618 23:31:58.928911 6467 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260618 23:31:58.930317 6482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:58.930636 6467 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:58.930758 6467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "2270c132876245d0b99153c4cc9f43db"
format_stamp: "Formatted at 2026-06-18 23:31:58 on dist-test-slave-6224"
server_key: "502461623d7c5713d8f2f762bc283649"
server_key_iv: "9348ca0cdcc5a1ac138c31801b4a871a"
server_key_version: "encryptionkey@0"
I20260618 23:31:58.930871 6467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:58.945375 6467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:58.947944 6467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:58.948073 6467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:58.952939 6467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43955
I20260618 23:31:58.953083 6534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43955 every 8 connection(s)
I20260618 23:31:58.953428 6467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:31:58.954483 6535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:58.957162 6535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db: Bootstrap starting.
I20260618 23:31:58.957778 6535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:58.958099 6535 log.cc:824] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:58.959529 6535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db: No bootstrap required, opened a new log
I20260618 23:31:58.960988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 6467
I20260618 23:31:58.961124 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:31:58.961339 392 external_mini_cluster.cc:1428] Setting key 7a0e4b4817567d39f2d8dd4896021c63
I20260618 23:31:58.961786 6535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } }
I20260618 23:31:58.961982 6535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:58.962019 6535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2270c132876245d0b99153c4cc9f43db, State: Initialized, Role: FOLLOWER
I20260618 23:31:58.962257 6535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } }
I20260618 23:31:58.962363 6535 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:31:58.962399 6535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:31:58.962438 6535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [term 0 FOLLOWER]: Advancing to term 1
Jun 18 23:31:58 dist-test-slave-6224 krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260618 23:31:58.969092 6535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } }
I20260618 23:31:58.969262 6535 leader_election.cc:302] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2270c132876245d0b99153c4cc9f43db; no voters:
I20260618 23:31:58.969563 6535 leader_election.cc:288] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:31:58.969707 6540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:31:58.970278 6540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [term 1 LEADER]: Becoming Leader. State: Replica: 2270c132876245d0b99153c4cc9f43db, State: Running, Role: LEADER
I20260618 23:31:58.969870 6535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:31:58.970482 6540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } }
I20260618 23:31:58.971338 6541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2270c132876245d0b99153c4cc9f43db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } }
I20260618 23:31:58.971465 6541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:58.971730 6542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2270c132876245d0b99153c4cc9f43db. Latest consensus state: current_term: 1 leader_uuid: "2270c132876245d0b99153c4cc9f43db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } }
I20260618 23:31:58.972020 6542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:58.972095 6546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:31:58.972664 6546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:31:58.973896 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:58.962011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.0.98.62:43955)
0618 23:31:58.962235 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:58.962246 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:58.962297 (+ 51us) server_negotiation.cc:407] Connection header received
0618 23:31:58.962441 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:58.962450 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:58.962519 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:58.962714 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:58.963476 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:58.964562 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:58.965370 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:58.965789 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:58.965881 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:58.968646 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:58.968650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:58.968651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:58.968663 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:58.971098 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:58.971717 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:58.971721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:58.971722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:58.971869 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:58.972250 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:58.972251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:58.972252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:58.972666 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:58.972843 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:58.973388 (+ 545us) server_negotiation.cc:299] Negotiation successful
0618 23:31:58.973587 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":117,"threads_started":1}
I20260618 23:31:58.975330 6546 catalog_manager.cc:1347] Generated new cluster ID: b45c6bc9c18d4dc7bb5d96301b19a73c
I20260618 23:31:58.975400 6546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:31:58.998975 6546 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:31:58.999778 6546 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:31:59.011065 6546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db: Generated new TSK 0
I20260618 23:31:59.011307 6546 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260618 23:31:59.064157 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43955
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:59.155779 6561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:59.156030 6561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:59.156056 6561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:59.157388 6561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:59.157445 6561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:59.157476 6561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:31:59.159094 6561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:59.159399 6561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:59.159714 6561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:59.165469 6566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:59.165469 6569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:59.165570 6567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:59.165618 6561 server_base.cc:1034] running on GCE node
I20260618 23:31:59.166035 6561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:59.166338 6561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:59.167512 6561 hybrid_clock.cc:648] HybridClock initialized: now 1781825519167489 us; error 47 us; skew 500 ppm
Jun 18 23:31:59 dist-test-slave-6224 krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825519, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:59.170840 6561 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260618 23:31:59.171386 6561 webserver.cc:460] Webserver started at http://127.0.98.1:36425/ using document root <none> and password file <none>
I20260618 23:31:59.171705 6561 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:59.171758 6561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:59.171871 6561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:59.173060 6561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "e2015d1fccc3456eae43d83da8ba816a"
format_stamp: "Formatted at 2026-06-18 23:31:59 on dist-test-slave-6224"
server_key: "d30059e8c9bf564c1ed1f5f6af447ac6"
server_key_iv: "96160827797105a0bc88ad44d9aaceaa"
server_key_version: "encryptionkey@0"
I20260618 23:31:59.173445 6561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e2015d1fccc3456eae43d83da8ba816a"
format_stamp: "Formatted at 2026-06-18 23:31:59 on dist-test-slave-6224"
server_key: "d30059e8c9bf564c1ed1f5f6af447ac6"
server_key_iv: "96160827797105a0bc88ad44d9aaceaa"
server_key_version: "encryptionkey@0"
I20260618 23:31:59.174908 6561 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:31:59.175771 6576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:59.175995 6561 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:59.176082 6561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "e2015d1fccc3456eae43d83da8ba816a"
format_stamp: "Formatted at 2026-06-18 23:31:59 on dist-test-slave-6224"
server_key: "d30059e8c9bf564c1ed1f5f6af447ac6"
server_key_iv: "96160827797105a0bc88ad44d9aaceaa"
server_key_version: "encryptionkey@0"
I20260618 23:31:59.176144 6561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:59.201893 6561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:59.204022 6561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:59.204146 6561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:59.204349 6561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:59.204698 6561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:59.204738 6561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:59.204761 6561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:59.204800 6561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:59.211649 6561 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32993
I20260618 23:31:59.211798 6689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32993 every 8 connection(s)
I20260618 23:31:59.212204 6561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
Jun 18 23:31:59 dist-test-slave-6224 krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825519, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260618 23:31:59.220717 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 6561
I20260618 23:31:59.220849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:31:59.221048 392 external_mini_cluster.cc:1428] Setting key f92a73c2e3957c6634fbdfdc856e50ec
I20260618 23:31:59.223347 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:59.213215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45957 (local address 127.0.98.62:43955)
0618 23:31:59.213318 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:59.213321 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:59.213621 (+ 300us) server_negotiation.cc:407] Connection header received
0618 23:31:59.214148 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:59.214149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:59.214165 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:59.214236 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:59.215237 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.215826 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.216893 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.217106 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.217182 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:59.219861 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:59.219864 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:59.219864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:59.219874 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:59.221503 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:59.222166 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:59.222167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:59.222167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:59.222221 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:59.222547 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:59.222547 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:59.222548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:59.222663 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:59.222724 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:59.223184 (+ 460us) server_negotiation.cc:299] Negotiation successful
0618 23:31:59.223275 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 23:31:59.223687 6692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:59.213181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.1:45957)
0618 23:31:59.213458 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:59.213502 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:59.214030 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:59.214298 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:59.214302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:59.214915 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:59.215123 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:59.215127 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.215923 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.215924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:59.216769 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:59.216771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.217235 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.217236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:59.217363 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:59.217977 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:59.217986 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:59.219743 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:59.221643 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:59.221650 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:59.221652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:59.222078 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:59.222302 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:59.222303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:59.222303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:59.222473 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:59.222739 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:59.222745 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:59.222963 (+ 218us) client_negotiation.cc:769] Sending connection context
0618 23:31:59.223094 (+ 131us) client_negotiation.cc:240] Negotiation successful
0618 23:31:59.223320 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":138,"threads_started":1}
I20260618 23:31:59.224261 6690 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43955
I20260618 23:31:59.224373 6690 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:59.224623 6690 heartbeater.cc:511] Master 127.0.98.62:43955 requested a full tablet report, sending...
I20260618 23:31:59.225265 6499 ts_manager.cc:194] Registered new tserver with Master: e2015d1fccc3456eae43d83da8ba816a (127.0.98.1:32993)
I20260618 23:31:59.225945 6499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45957
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260618 23:31:59.250330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43955
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:59.336102 6695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:59.336320 6695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:59.336344 6695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:59.337461 6695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:59.337504 6695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:59.337530 6695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:31:59.338900 6695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:59.339094 6695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:59.339277 6695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:59.344882 6701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:59.344902 6700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:59.344924 6695 server_base.cc:1034] running on GCE node
W20260618 23:31:59.344923 6703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:59.345587 6695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:59.345849 6695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:59.347095 6695 hybrid_clock.cc:648] HybridClock initialized: now 1781825519347070 us; error 47 us; skew 500 ppm
Jun 18 23:31:59 dist-test-slave-6224 krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825519, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:59.349871 6695 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260618 23:31:59.350360 6695 webserver.cc:460] Webserver started at http://127.0.98.2:39821/ using document root <none> and password file <none>
I20260618 23:31:59.350598 6695 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:59.350646 6695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:59.350800 6695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:59.351958 6695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "8232a14a2ef3499ea04b5f76d538cf83"
format_stamp: "Formatted at 2026-06-18 23:31:59 on dist-test-slave-6224"
server_key: "8476eb42989d9b345fe25a3d61a966e7"
server_key_iv: "43b1880f705e67ef3339d194486d79db"
server_key_version: "encryptionkey@0"
I20260618 23:31:59.352336 6695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8232a14a2ef3499ea04b5f76d538cf83"
format_stamp: "Formatted at 2026-06-18 23:31:59 on dist-test-slave-6224"
server_key: "8476eb42989d9b345fe25a3d61a966e7"
server_key_iv: "43b1880f705e67ef3339d194486d79db"
server_key_version: "encryptionkey@0"
I20260618 23:31:59.353761 6695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:31:59.354637 6710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:59.354861 6695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:59.355003 6695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "8232a14a2ef3499ea04b5f76d538cf83"
format_stamp: "Formatted at 2026-06-18 23:31:59 on dist-test-slave-6224"
server_key: "8476eb42989d9b345fe25a3d61a966e7"
server_key_iv: "43b1880f705e67ef3339d194486d79db"
server_key_version: "encryptionkey@0"
I20260618 23:31:59.355082 6695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:59.364135 6695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:59.366276 6695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:59.366436 6695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:59.366664 6695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:59.366993 6695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:59.367051 6695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:59.367096 6695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:59.367123 6695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:59.374326 6695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35403
I20260618 23:31:59.374440 6823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35403 every 8 connection(s)
I20260618 23:31:59.374755 6695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:31:59.376096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 6695
I20260618 23:31:59.376212 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:31:59.376434 392 external_mini_cluster.cc:1428] Setting key ae5cc168b2b7b11e75c870174b834ccd
Jun 18 23:31:59 dist-test-slave-6224 krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825519, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260618 23:31:59.387441 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:59.375696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58319 (local address 127.0.98.62:43955)
0618 23:31:59.375761 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:59.375763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:59.376146 (+ 383us) server_negotiation.cc:407] Connection header received
0618 23:31:59.376960 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:59.376962 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:59.376974 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:59.377024 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:59.378339 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.379251 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.380426 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.380821 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.380911 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:59.383744 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:59.383746 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:59.383746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:59.383757 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:59.385768 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:59.386250 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:59.386252 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:59.386253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:59.386320 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:59.386601 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:59.386602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:59.386602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:59.386788 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:59.386875 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:59.387176 (+ 301us) server_negotiation.cc:299] Negotiation successful
0618 23:31:59.387297 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 23:31:59.387475 6826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:59.375735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.2:58319)
0618 23:31:59.375976 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:59.376031 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:59.376781 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:59.377092 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:59.377096 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:59.377950 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:59.378231 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:59.378237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.379348 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.379349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:59.380339 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:59.380341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.380918 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.380918 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:59.381027 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:59.381681 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:59.381689 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:59.383639 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:59.385875 (+ 2236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:59.385881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:59.385882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:59.386133 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:59.386404 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:59.386405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:59.386405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:59.386503 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:59.386868 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:59.386871 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:59.386987 (+ 116us) client_negotiation.cc:769] Sending connection context
0618 23:31:59.387097 (+ 110us) client_negotiation.cc:240] Negotiation successful
0618 23:31:59.387307 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":120,"threads_started":1}
I20260618 23:31:59.388202 6824 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43955
I20260618 23:31:59.388345 6824 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:59.388559 6824 heartbeater.cc:511] Master 127.0.98.62:43955 requested a full tablet report, sending...
I20260618 23:31:59.389056 6499 ts_manager.cc:194] Registered new tserver with Master: 8232a14a2ef3499ea04b5f76d538cf83 (127.0.98.2:35403)
I20260618 23:31:59.389573 6499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260618 23:31:59.404362 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43955
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:59.497880 6829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:59.498183 6829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:59.498214 6829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:59.499675 6829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:59.499740 6829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:59.499775 6829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:31:59.501791 6829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43955
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:59.502074 6829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:59.502321 6829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:59.508524 6835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:59.508596 6837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:59.508533 6834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:59.508920 6829 server_base.cc:1034] running on GCE node
I20260618 23:31:59.509146 6829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:59.509406 6829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:59.510576 6829 hybrid_clock.cc:648] HybridClock initialized: now 1781825519510558 us; error 47 us; skew 500 ppm
Jun 18 23:31:59 dist-test-slave-6224 krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825519, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:59.513804 6829 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260618 23:31:59.514415 6829 webserver.cc:460] Webserver started at http://127.0.98.3:43525/ using document root <none> and password file <none>
I20260618 23:31:59.514689 6829 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:59.514762 6829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:59.514930 6829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:59.516290 6829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5"
format_stamp: "Formatted at 2026-06-18 23:31:59 on dist-test-slave-6224"
server_key: "4504b0eb8b92c9578c973b0bd580d69f"
server_key_iv: "3f288b438553d5a382e7e9711df36ac3"
server_key_version: "encryptionkey@0"
I20260618 23:31:59.516700 6829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5"
format_stamp: "Formatted at 2026-06-18 23:31:59 on dist-test-slave-6224"
server_key: "4504b0eb8b92c9578c973b0bd580d69f"
server_key_iv: "3f288b438553d5a382e7e9711df36ac3"
server_key_version: "encryptionkey@0"
I20260618 23:31:59.518353 6829 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:31:59.519343 6844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:59.519672 6829 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:59.519794 6829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5"
format_stamp: "Formatted at 2026-06-18 23:31:59 on dist-test-slave-6224"
server_key: "4504b0eb8b92c9578c973b0bd580d69f"
server_key_iv: "3f288b438553d5a382e7e9711df36ac3"
server_key_version: "encryptionkey@0"
I20260618 23:31:59.519897 6829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:59.537184 6829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:59.539804 6829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:59.540014 6829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:59.540365 6829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:31:59.540787 6829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:31:59.540854 6829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:59.540882 6829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:31:59.540895 6829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:59.548458 6829 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37473
I20260618 23:31:59.548506 6957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37473 every 8 connection(s)
I20260618 23:31:59.548884 6829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:31:59.550318 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 6829
I20260618 23:31:59.550446 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:31:59.550674 392 external_mini_cluster.cc:1428] Setting key 6f2e9ac1a1b8e37da6bd1121ffaafcb5
Jun 18 23:31:59 dist-test-slave-6224 krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825519, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260618 23:31:59.560462 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:59.549799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42467 (local address 127.0.98.62:43955)
0618 23:31:59.549940 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:59.549943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:59.550162 (+ 219us) server_negotiation.cc:407] Connection header received
0618 23:31:59.550854 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:59.550856 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:59.550872 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:59.550948 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:59.552389 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.552929 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.554128 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.554363 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.554440 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:59.557113 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:59.557114 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:59.557115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:59.557122 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:59.558717 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:59.559236 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:59.559237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:59.559237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:59.559297 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:59.559683 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:59.559684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:59.559684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:59.559834 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:59.559914 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:59.560280 (+ 366us) server_negotiation.cc:299] Negotiation successful
0618 23:31:59.560367 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 23:31:59.560565 6960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:59.549758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.3:42467)
0618 23:31:59.549986 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:59.550041 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:59.550714 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:59.551047 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:59.551052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:59.552004 (+ 952us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:59.552291 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:59.552296 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.553048 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.553049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:59.554006 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:59.554008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.554467 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.554468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:59.554582 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:59.555242 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:59.555250 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:59.556983 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:59.558850 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:59.558858 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:59.558868 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:59.559156 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:59.559388 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:59.559389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:59.559390 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:59.559503 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:59.559945 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:59.559951 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:59.560092 (+ 141us) client_negotiation.cc:769] Sending connection context
0618 23:31:59.560209 (+ 117us) client_negotiation.cc:240] Negotiation successful
0618 23:31:59.560355 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":108,"threads_started":1}
I20260618 23:31:59.561403 6958 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43955
I20260618 23:31:59.561573 6958 heartbeater.cc:464] Registering TS with master...
I20260618 23:31:59.561827 6958 heartbeater.cc:511] Master 127.0.98.62:43955 requested a full tablet report, sending...
I20260618 23:31:59.562377 6498 ts_manager.cc:194] Registered new tserver with Master: 0a88d6f8b7bf48ffb3dd2bc35d448cd5 (127.0.98.3:37473)
I20260618 23:31:59.562952 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:31:59.562999 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42467
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260618 23:31:59.589968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36757
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43955,127.0.98.61:36757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:59.680083 6963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:59.680358 6963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:59.680397 6963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:59.681710 6963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:31:59.681778 6963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:59.681803 6963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:59.681833 6963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:31:59.681866 6963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:31:59.683570 6963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43955,127.0.98.61:36757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
W20260618 23:31:59.683732 6963 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:43955,127.0.98.61:36757'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260618 23:31:59.683923 6963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:59.684209 6963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:59.690052 6969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:59.690186 6971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:59.690261 6963 server_base.cc:1034] running on GCE node
W20260618 23:31:59.690052 6968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:59.690663 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:59.690989 6963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:59.692169 6963 hybrid_clock.cc:648] HybridClock initialized: now 1781825519692148 us; error 42 us; skew 500 ppm
Jun 18 23:31:59 dist-test-slave-6224 krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825519, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:59.695315 6963 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260618 23:31:59.696059 6963 webserver.cc:460] Webserver started at http://127.0.98.61:44499/ using document root <none> and password file <none>
I20260618 23:31:59.696442 6963 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:59.696525 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:59.696700 6963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:31:59.698026 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/data/instance:
uuid: "22b5aed081f942d599a7192cba8c03a8"
format_stamp: "Formatted at 2026-06-18 23:31:59 on dist-test-slave-6224"
server_key: "7ba2de64139953e71e981a78f52ddfbf"
server_key_iv: "4a6409297f6e3844106a1aa8cc118a36"
server_key_version: "encryptionkey@0"
I20260618 23:31:59.698441 6963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/wal/instance:
uuid: "22b5aed081f942d599a7192cba8c03a8"
format_stamp: "Formatted at 2026-06-18 23:31:59 on dist-test-slave-6224"
server_key: "7ba2de64139953e71e981a78f52ddfbf"
server_key_iv: "4a6409297f6e3844106a1aa8cc118a36"
server_key_version: "encryptionkey@0"
I20260618 23:31:59.700238 6963 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260618 23:31:59.701538 6978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:59.701756 6963 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:31:59.701906 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/wal
uuid: "22b5aed081f942d599a7192cba8c03a8"
format_stamp: "Formatted at 2026-06-18 23:31:59 on dist-test-slave-6224"
server_key: "7ba2de64139953e71e981a78f52ddfbf"
server_key_iv: "4a6409297f6e3844106a1aa8cc118a36"
server_key_version: "encryptionkey@0"
I20260618 23:31:59.702020 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:59.721031 6963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:59.723380 6963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:31:59.723523 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:59.727838 6963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36757
I20260618 23:31:59.727938 7030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36757 every 8 connection(s)
I20260618 23:31:59.728284 6963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/data/info.pb
I20260618 23:31:59.729429 7031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:31:59.731158 7031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:31:59.735491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 6963
Jun 18 23:31:59 dist-test-slave-6224 krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825519, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 18 23:31:59 dist-test-slave-6224 krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260618 23:31:59.741853 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:59.731991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48276 (local address 127.0.98.62:43955)
0618 23:31:59.732087 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:59.732090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:59.732387 (+ 297us) server_negotiation.cc:407] Connection header received
0618 23:31:59.733382 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:59.733384 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:59.733396 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:59.733465 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:59.734116 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.734747 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.735946 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.736245 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.736325 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:59.738843 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:59.738844 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:59.738844 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:59.738851 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:59.740370 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:59.740795 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:59.740796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:59.740797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:59.740856 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:59.741101 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:59.741102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:59.741103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:59.741307 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:59.741383 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:59.741678 (+ 295us) server_negotiation.cc:299] Negotiation successful
0618 23:31:59.741760 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 23:31:59.742058 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:59.731950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.0.1:48276)
0618 23:31:59.732217 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:59.732278 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:59.733242 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:59.733550 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:59.733554 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:59.733798 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:59.734029 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:59.734033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.734827 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.734828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:59.735835 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:59.735837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.736325 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.736326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:59.736491 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:59.737123 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:59.737131 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:59.738747 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:59.740484 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:59.740491 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:59.740493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:59.740720 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:59.740925 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:59.740926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:59.740926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:59.741035 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:59.741384 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:59.741391 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:59.741538 (+ 147us) client_negotiation.cc:769] Sending connection context
0618 23:31:59.741658 (+ 120us) client_negotiation.cc:240] Negotiation successful
0618 23:31:59.741799 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":100,"threads_started":1}
I20260618 23:31:59.742733 7031 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:31:59.745402 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:59.736127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35384 (local address 127.0.98.61:36757)
0618 23:31:59.736442 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:59.736446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:59.736464 (+ 18us) server_negotiation.cc:407] Connection header received
0618 23:31:59.736485 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:59.736487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:59.736505 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:59.736601 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:59.737364 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.738191 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.739049 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.739441 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.739518 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:59.741929 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:59.741940 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:59.741940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:59.741950 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:59.744177 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:59.744580 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:59.744583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:59.744583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:59.744651 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:59.744875 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:59.744876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:59.744876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:59.745012 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:59.745159 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:59.745236 (+ 77us) server_negotiation.cc:299] Negotiation successful
0618 23:31:59.745299 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":103,"threads_started":1}
Jun 18 23:31:59 dist-test-slave-6224 krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825519, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260618 23:31:59.750685 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:59.743031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36757 (local address 127.0.0.1:35388)
0618 23:31:59.743246 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:59.743258 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:59.743305 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:59.743466 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:59.743467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:59.743672 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:59.743720 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:59.743721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.744619 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.744620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:59.745329 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:59.745333 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.745877 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.745877 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:59.745946 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:59.746347 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:59.746351 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:59.747517 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:59.749627 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:59.749628 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:59.749628 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:59.749800 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:59.750020 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:59.750021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:59.750021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:59.750075 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:59.750330 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:59.750331 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:59.750369 (+ 38us) client_negotiation.cc:769] Sending connection context
0618 23:31:59.750440 (+ 71us) client_negotiation.cc:240] Negotiation successful
0618 23:31:59.750549 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197}
I20260618 23:31:59.750721 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:59.743101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35388 (local address 127.0.98.61:36757)
0618 23:31:59.743301 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:59.743303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:59.743313 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:31:59.743345 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:59.743345 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:59.743358 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:59.743453 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:59.743808 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.744507 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.745517 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.745790 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.745861 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:59.747649 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:59.747650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:59.747651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:59.747657 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:59.749537 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:59.749886 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:59.749887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:59.749887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:59.749945 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:59.750144 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:59.750145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:59.750146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:59.750263 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:59.750344 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:59.750463 (+ 119us) server_negotiation.cc:299] Negotiation successful
0618 23:31:59.750597 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":171,"thread_start_us":98,"threads_started":1}
I20260618 23:31:59.752436 7031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8: Bootstrap starting.
I20260618 23:31:59.753129 7031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8: Neither blocks nor log segments found. Creating new log.
I20260618 23:31:59.753384 7031 log.cc:824] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8: Log is configured to *not* fsync() on all Append() calls
I20260618 23:31:59.754223 7031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8: No bootstrap required, opened a new log
I20260618 23:31:59.756125 7031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } }
I20260618 23:31:59.756275 7031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:31:59.756300 7031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22b5aed081f942d599a7192cba8c03a8, State: Initialized, Role: FOLLOWER
I20260618 23:31:59.756445 7031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } }
I20260618 23:31:59.756791 7031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:31:59.756770 7038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } } }
I20260618 23:31:59.757025 7038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [sys.catalog]: This master's current role is: FOLLOWER
W20260618 23:31:59.758939 7049 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:31:59.759016 7049 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:31:59.761158 6963 master_runner.cc:428] Detected that this master 22b5aed081f942d599a7192cba8c03a8 is joining an existing cluster
I20260618 23:31:59.761229 6963 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:36757
I20260618 23:31:59.770294 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:59.762443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48278 (local address 127.0.98.62:43955)
0618 23:31:59.762505 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:59.762508 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:59.762856 (+ 348us) server_negotiation.cc:407] Connection header received
0618 23:31:59.762930 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:59.762931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:59.762946 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:59.762988 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:59.763659 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.764529 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.765417 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.765756 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.765835 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:59.767218 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:59.767219 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:59.767219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:59.767227 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:59.768880 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:59.769321 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:59.769322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:59.769323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:59.769383 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:59.769627 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:59.769628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:59.769628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:59.769763 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:59.769828 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:59.770113 (+ 285us) server_negotiation.cc:299] Negotiation successful
0618 23:31:59.770200 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260618 23:31:59.770390 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:59.762572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.0.1:48278)
0618 23:31:59.762756 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:59.762770 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:59.762845 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:59.763062 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:59.763063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:59.763397 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:59.763516 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:59.763520 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.764629 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.764630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:59.765305 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:59.765306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.765895 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.765897 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:59.765986 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:59.766646 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:59.766650 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:59.767129 (+ 479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:59.768979 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:59.768980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:59.768980 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:59.769209 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:59.769488 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:59.769489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:59.769489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:59.769552 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:59.769897 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:59.769898 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:59.769956 (+ 58us) client_negotiation.cc:769] Sending connection context
0618 23:31:59.770080 (+ 124us) client_negotiation.cc:240] Negotiation successful
0618 23:31:59.770265 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":8,"thread_start_us":120,"threads_started":1}
I20260618 23:31:59.771754 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:59.762394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36757 (local address 127.0.0.1:35396)
0618 23:31:59.762756 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:59.762770 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:59.762853 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:59.763026 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:59.763027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:59.763394 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:59.763523 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:59.763524 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.765506 (+ 1982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.765507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:59.766218 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:59.766220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.766735 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.766736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:59.766813 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:59.767471 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:59.767475 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:59.768291 (+ 816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:59.770553 (+ 2262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:59.770554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:59.770555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:59.770757 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:59.771009 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:59.771010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:59.771010 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:59.771064 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:59.771308 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:59.771308 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:59.771344 (+ 36us) client_negotiation.cc:769] Sending connection context
0618 23:31:59.771458 (+ 114us) client_negotiation.cc:240] Negotiation successful
0618 23:31:59.771631 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":3,"thread_start_us":97,"threads_started":1}
I20260618 23:31:59.771998 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:59.762410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35396 (local address 127.0.98.61:36757)
0618 23:31:59.762463 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:59.762465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:59.762894 (+ 429us) server_negotiation.cc:407] Connection header received
0618 23:31:59.762921 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:59.762923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:59.762937 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:59.763011 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:59.764654 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.765403 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.766320 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.766641 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.766724 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:59.768391 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:59.768393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:59.768393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:59.768401 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:59.770453 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:59.770840 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:59.770841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:59.770841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:59.770938 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:59.771134 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:59.771135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:59.771136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:59.771245 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:59.771318 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:59.771511 (+ 193us) server_negotiation.cc:299] Negotiation successful
0618 23:31:59.771732 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":22}
Jun 18 23:31:59 dist-test-slave-6224 krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825518, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260618 23:31:59.781759 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:59.774042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35412 (local address 127.0.98.61:36757)
0618 23:31:59.774105 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:59.774107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:59.774274 (+ 167us) server_negotiation.cc:407] Connection header received
0618 23:31:59.774366 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:59.774367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:59.774377 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:59.774432 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:59.774870 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.775341 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.776178 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.776374 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.776455 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:59.778869 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:59.778870 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:59.778870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:59.778876 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:59.780476 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:59.780994 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:59.780995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:59.780995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:59.781044 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:59.781270 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:59.781270 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:59.781271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:59.781363 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:59.781414 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:59.781638 (+ 224us) server_negotiation.cc:299] Negotiation successful
0618 23:31:59.781697 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 23:31:59.781910 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:59.773964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36757 (local address 127.0.0.1:35412)
0618 23:31:59.774177 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:59.774211 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:59.774301 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:59.774437 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:59.774440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:59.774731 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:59.774781 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:59.774783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.775419 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.775420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:59.776079 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:59.776082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.776462 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.776463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:59.776555 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:59.777023 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:59.777028 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:59.778766 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:59.780563 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:59.780571 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:59.780573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:59.780889 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:59.781120 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:59.781121 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:59.781122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:59.781200 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:59.781432 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:59.781440 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:59.781512 (+ 72us) client_negotiation.cc:769] Sending connection context
0618 23:31:59.781618 (+ 106us) client_negotiation.cc:240] Negotiation successful
0618 23:31:59.781771 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":118,"threads_started":1}
I20260618 23:31:59.782611 6498 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:36757: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "22b5aed081f942d599a7192cba8c03a8"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 36757
}
attrs {
promote: true
}
}
dest_uuid: "2270c132876245d0b99153c4cc9f43db"
cas_config_opid_index: -1
I20260618 23:31:59.782965 6498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: true } }
I20260618 23:31:59.783540 7058 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 22b5aed081f942d599a7192cba8c03a8 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: true } } }
I20260618 23:31:59.783910 7005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:31:59.784684 7058 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:36757
I20260618 23:31:59.784852 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2270c132876245d0b99153c4cc9f43db" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: true } } }
I20260618 23:31:59.784933 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [sys.catalog]: This master's current role is: LEADER
I20260618 23:31:59.784931 6963 master.cc:537] Master@127.0.98.61:36757 shutting down...
I20260618 23:31:59.784981 7005 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [term 1 FOLLOWER]: Refusing update from remote peer 2270c132876245d0b99153c4cc9f43db: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260618 23:31:59.785298 7058 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260618 23:31:59.785943 6485 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260618 23:31:59.786360 6485 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db -> Peer 22b5aed081f942d599a7192cba8c03a8 (127.0.98.61:36757): Couldn't send request to peer 22b5aed081f942d599a7192cba8c03a8. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260618 23:31:59.788360 6963 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:31:59.788453 6963 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:31:59.788511 6963 tablet_replica.cc:330] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8: stopping tablet replica
I20260618 23:31:59.791242 6963 master.cc:552] Master@127.0.98.61:36757 shutdown complete.
I20260618 23:31:59.791306 6963 master_runner.cc:305] Clearing existing system tablet
I20260618 23:31:59.791461 6963 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:59.791515 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:59.792325 6963 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:31:59.792930 7063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:59.793198 6963 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:59.793272 6963 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:31:59.793298 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/wal
uuid: "22b5aed081f942d599a7192cba8c03a8"
format_stamp: "Formatted at 2026-06-18 23:31:59 on dist-test-slave-6224"
server_key: "7ba2de64139953e71e981a78f52ddfbf"
server_key_iv: "4a6409297f6e3844106a1aa8cc118a36"
server_key_version: "encryptionkey@0"
I20260618 23:31:59.794219 6963 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:31:59.795825 6963 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260618 23:31:59.795904 6963 log.cc:1196] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260618 23:31:59.796123 6963 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8: Deleting consensus metadata
I20260618 23:31:59.796252 6963 master_runner.cc:315] Copying system tablet from 127.0.98.62:43955
I20260618 23:31:59.796970 6963 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:43955
I20260618 23:31:59.804064 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:59.797365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48288 (local address 127.0.98.62:43955)
0618 23:31:59.797409 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:59.797411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:59.797498 (+ 87us) server_negotiation.cc:407] Connection header received
0618 23:31:59.797582 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:59.797582 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:59.797591 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:59.797646 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:59.798047 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.798534 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.799828 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.800015 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.800089 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:59.801287 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:59.801288 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:59.801288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:59.801294 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:59.802828 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:59.803245 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:59.803246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:59.803246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:59.803294 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:59.803504 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:59.803505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:59.803505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:59.803674 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:59.803733 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:59.803937 (+ 204us) server_negotiation.cc:299] Negotiation successful
0618 23:31:59.803997 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260618 23:31:59.804322 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:59.797258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.0.1:48288)
0618 23:31:59.797407 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0618 23:31:59.797421 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:31:59.797519 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:31:59.797659 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:31:59.797660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:31:59.797895 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:31:59.797958 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:59.797960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.798620 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.798621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:59.799660 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:31:59.799662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.800139 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.800141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:31:59.800212 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:59.800721 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:31:59.800727 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:31:59.801183 (+ 456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:31:59.802919 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:59.802920 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:59.802921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:59.803161 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:59.803366 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:31:59.803367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:31:59.803367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:31:59.803436 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:31:59.803746 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:31:59.803748 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:31:59.803800 (+ 52us) client_negotiation.cc:769] Sending connection context
0618 23:31:59.803881 (+ 81us) client_negotiation.cc:240] Negotiation successful
0618 23:31:59.803994 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260618 23:31:59.804709 6519 tablet_copy_service.cc:140] P 2270c132876245d0b99153c4cc9f43db: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 22b5aed081f942d599a7192cba8c03a8 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:48288)
I20260618 23:31:59.804808 6519 tablet_copy_service.cc:161] P 2270c132876245d0b99153c4cc9f43db: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 22b5aed081f942d599a7192cba8c03a8 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:48288: session id = 22b5aed081f942d599a7192cba8c03a8-00000000000000000000000000000000
I20260618 23:31:59.805423 6519 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db: Tablet Copy: opened 0 blocks and 1 log segments
I20260618 23:31:59.806121 6963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260618 23:31:59.807211 6963 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8: tablet copy: Starting download of 0 data blocks...
I20260618 23:31:59.807389 6963 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8: tablet copy: Starting download of 1 WAL segments...
I20260618 23:31:59.808446 6963 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260618 23:31:59.809289 6519 tablet_copy_service.cc:342] P 2270c132876245d0b99153c4cc9f43db: Request end of tablet copy session 22b5aed081f942d599a7192cba8c03a8-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:48288
I20260618 23:31:59.809353 6519 tablet_copy_service.cc:433] P 2270c132876245d0b99153c4cc9f43db: ending tablet copy session 22b5aed081f942d599a7192cba8c03a8-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 22b5aed081f942d599a7192cba8c03a8
I20260618 23:31:59.812352 6963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:31:59.813731 7070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:31:59.813731 7071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:59.813973 6963 server_base.cc:1034] running on GCE node
W20260618 23:31:59.813988 7073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:31:59.814247 6963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:31:59.814414 6963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:31:59.815702 6963 hybrid_clock.cc:648] HybridClock initialized: now 1781825519815696 us; error 27 us; skew 500 ppm
Jun 18 23:31:59 dist-test-slave-6224 krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825519, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:31:59.817207 6963 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260618 23:31:59.817551 6963 webserver.cc:460] Webserver started at http://127.0.98.61:36519/ using document root <none> and password file <none>
I20260618 23:31:59.817677 6963 fs_manager.cc:359] Metadata directory not provided
I20260618 23:31:59.817734 6963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:31:59.818408 6963 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260618 23:31:59.818979 7080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:31:59.819142 6963 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260618 23:31:59.819193 6963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/wal
uuid: "22b5aed081f942d599a7192cba8c03a8"
format_stamp: "Formatted at 2026-06-18 23:31:59 on dist-test-slave-6224"
server_key: "7ba2de64139953e71e981a78f52ddfbf"
server_key_iv: "4a6409297f6e3844106a1aa8cc118a36"
server_key_version: "encryptionkey@0"
I20260618 23:31:59.819267 6963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:31:59.838135 6963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:31:59.839905 6963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:31:59.843233 6963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36757
I20260618 23:31:59.843539 7132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36757 every 8 connection(s)
I20260618 23:31:59.843686 6963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-1/data/info.pb
I20260618 23:31:59.844486 7133 sys_catalog.cc:263] Verifying existing consensus state
I20260618 23:31:59.844941 7133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8: Bootstrap starting.
I20260618 23:31:59.848008 7133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:31:59.848281 7133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8: Bootstrap complete.
I20260618 23:31:59.848500 7133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: true } }
I20260618 23:31:59.848589 7133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 22b5aed081f942d599a7192cba8c03a8, State: Initialized, Role: LEARNER
I20260618 23:31:59.848634 7133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: true } }
I20260618 23:31:59.848825 7133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:31:59.848788 7137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: true } } }
I20260618 23:31:59.849993 7137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [sys.catalog]: This master's current role is: LEARNER
I20260618 23:31:59.850592 7148 catalog_manager.cc:1259] Loaded cluster ID: b45c6bc9c18d4dc7bb5d96301b19a73c
I20260618 23:31:59.850648 7148 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8: loading cluster ID for follower catalog manager: success
I20260618 23:31:59.851882 7148 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8: acquiring CA information for follower catalog manager: success
I20260618 23:31:59.852245 7148 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 23:31:59.855952 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:31:59.847508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35430 (local address 127.0.98.61:36757)
0618 23:31:59.847742 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0618 23:31:59.847744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:31:59.847753 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:31:59.847777 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:31:59.847777 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:31:59.847788 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:31:59.847876 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:31:59.848735 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.849820 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.850891 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:31:59.851240 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:31:59.851309 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:31:59.852832 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:31:59.852834 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:31:59.852835 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:31:59.852845 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:31:59.854611 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:59.854957 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:59.854958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:59.854958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:59.855027 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:31:59.855262 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:31:59.855263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:31:59.855264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:31:59.855410 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:31:59.855528 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0618 23:31:59.855694 (+ 166us) server_negotiation.cc:299] Negotiation successful
0618 23:31:59.855844 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":159,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260618 23:31:59.886442 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40637
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43955,127.0.98.61:36757,127.0.98.60:40637
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:31:59.992189 7151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:31:59.992410 7151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:31:59.992434 7151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:31:59.993670 7151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:31:59.993721 7151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:31:59.993736 7151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:31:59.993749 7151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:31:59.993762 7151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:31:59.995213 7151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43955,127.0.98.61:36757,127.0.98.60:40637
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:31:59.995406 7151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:31:59.995647 7151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:00.000654 7156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:00.000713 7159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:00.000713 7157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:00.001274 7151 server_base.cc:1034] running on GCE node
I20260618 23:32:00.001629 7151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:00.002031 7151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:00.003237 7151 hybrid_clock.cc:648] HybridClock initialized: now 1781825520003176 us; error 82 us; skew 500 ppm
Jun 18 23:32:00 dist-test-slave-6224 krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825520, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:32:00.007408 7151 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260618 23:32:00.008124 7151 webserver.cc:460] Webserver started at http://127.0.98.60:34175/ using document root <none> and password file <none>
I20260618 23:32:00.008410 7151 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:00.008486 7151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:00.008667 7151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:32:00.009966 7151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/data/instance:
uuid: "34f7483731cd4088aad3cbc7663639fe"
format_stamp: "Formatted at 2026-06-18 23:32:00 on dist-test-slave-6224"
server_key: "79d441fce6bf5b6f73e35bb8723f71e2"
server_key_iv: "8d32f6451e37d80a1aaecb45a523d087"
server_key_version: "encryptionkey@0"
I20260618 23:32:00.010423 7151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/wal/instance:
uuid: "34f7483731cd4088aad3cbc7663639fe"
format_stamp: "Formatted at 2026-06-18 23:32:00 on dist-test-slave-6224"
server_key: "79d441fce6bf5b6f73e35bb8723f71e2"
server_key_iv: "8d32f6451e37d80a1aaecb45a523d087"
server_key_version: "encryptionkey@0"
I20260618 23:32:00.012277 7151 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260618 23:32:00.013623 7166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:00.013934 7151 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:00.014112 7151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/wal
uuid: "34f7483731cd4088aad3cbc7663639fe"
format_stamp: "Formatted at 2026-06-18 23:32:00 on dist-test-slave-6224"
server_key: "79d441fce6bf5b6f73e35bb8723f71e2"
server_key_iv: "8d32f6451e37d80a1aaecb45a523d087"
server_key_version: "encryptionkey@0"
I20260618 23:32:00.014245 7151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:00.038828 7151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:00.041342 7151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:00.041549 7151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:00.046664 7151 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40637
I20260618 23:32:00.046717 7218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40637 every 8 connection(s)
I20260618 23:32:00.047175 7151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/data/info.pb
I20260618 23:32:00.048403 7219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:00.051630 7219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:32:00.054049 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 7151
Jun 18 23:32:00 dist-test-slave-6224 krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825520, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 18 23:32:00 dist-test-slave-6224 krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260618 23:32:00.065774 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.052593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48292 (local address 127.0.98.62:43955)
0618 23:32:00.052693 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.052695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.052842 (+ 147us) server_negotiation.cc:407] Connection header received
0618 23:32:00.053507 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.053508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.053520 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.053624 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:00.054555 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.055341 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.057089 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.058157 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.058227 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.061190 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:00.061191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:00.061191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:00.061199 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:00.063886 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.064508 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.064510 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.064510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.064597 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.064943 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.064944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.064945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.065080 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:00.065192 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.065553 (+ 361us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.065660 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 23:32:00.066062 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.052494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.0.1:48292)
0618 23:32:00.052699 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.052745 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.053361 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.053690 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.053694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.054085 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:00.054420 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.054426 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.055474 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.055475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.056959 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.056962 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.058298 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.058299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.058436 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.059067 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:00.059076 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:00.061085 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:00.063990 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.063996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.063999 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.064349 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.064686 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.064687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.064688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.064823 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.065189 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:00.065204 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:00.065399 (+ 195us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.065530 (+ 131us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.065739 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":115,"threads_started":1}
I20260618 23:32:00.066318 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.054983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56478 (local address 127.0.98.60:40637)
0618 23:32:00.055237 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.055242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.055283 (+ 41us) server_negotiation.cc:407] Connection header received
0618 23:32:00.055314 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.055316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.055339 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.055470 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:00.056342 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.057682 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.059405 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.059895 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.059962 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.062896 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:00.062904 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:00.062904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:00.062911 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:00.064904 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.065242 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.065246 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.065246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.065306 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.065515 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.065516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.065517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.065675 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:00.065830 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.065901 (+ 71us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.065996 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":154,"threads_started":1}
I20260618 23:32:00.066835 7219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 23:32:00 dist-test-slave-6224 krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825520, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260618 23:32:00.074668 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.067189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36757 (local address 127.0.0.1:35434)
0618 23:32:00.067251 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.067263 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.067303 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.067498 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.067499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.067793 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:00.067862 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.067864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.068551 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.068552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.069129 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.069131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.069533 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.069534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.069622 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.070139 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:00.070145 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:00.071627 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:00.073617 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.073618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.073619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.073790 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.074050 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.074051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.074051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.074105 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.074406 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:00.074407 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:00.074450 (+ 43us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.074509 (+ 59us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.074596 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260618 23:32:00.074736 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.067258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35434 (local address 127.0.98.61:36757)
0618 23:32:00.067334 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.067337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.067344 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:32:00.067366 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.067367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.067377 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.067434 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:00.067945 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.068451 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.069225 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.069428 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.069489 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.071740 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:00.071742 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:00.071743 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:00.071750 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:00.073519 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.073893 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.073894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.073895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.073974 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.074193 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.074194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.074194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.074331 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:00.074398 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.074535 (+ 137us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.074622 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 23:32:00.075294 7219 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40637 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 23:32:00 dist-test-slave-6224 krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825520, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260618 23:32:00.082666 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.075748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40637 (local address 127.0.0.1:56480)
0618 23:32:00.075817 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.075831 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.075888 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.076013 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.076013 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.076147 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:00.076185 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.076187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.076946 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.076947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.077475 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.077476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.077883 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.077884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.077928 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.078347 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:00.078351 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:00.079782 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:00.081597 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.081598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.081599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.081764 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.082029 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.082030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.082030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.082087 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.082401 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:00.082402 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:00.082443 (+ 41us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.082514 (+ 71us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.082600 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260618 23:32:00.082840 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.075860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56480 (local address 127.0.98.60:40637)
0618 23:32:00.075907 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.075909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.075915 (+ 6us) server_negotiation.cc:407] Connection header received
0618 23:32:00.075949 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.075949 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.075959 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.076008 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:00.076298 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.076841 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.077573 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.077798 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.077868 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.079942 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:00.079946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:00.079946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:00.079957 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:00.081504 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.081877 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.081878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.081878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.081949 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.082151 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.082152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.082152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.082273 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:00.082337 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.082554 (+ 217us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.082659 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":27}
I20260618 23:32:00.084774 7219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe: Bootstrap starting.
I20260618 23:32:00.085650 7219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:00.085927 7219 log.cc:824] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe: Log is configured to *not* fsync() on all Append() calls
I20260618 23:32:00.086771 7219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe: No bootstrap required, opened a new log
I20260618 23:32:00.088259 7219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40637 } }
I20260618 23:32:00.088433 7219 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:00.088481 7219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f7483731cd4088aad3cbc7663639fe, State: Initialized, Role: FOLLOWER
I20260618 23:32:00.088627 7219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40637 } }
I20260618 23:32:00.088919 7219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:32:00.088976 7224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40637 } } }
I20260618 23:32:00.089149 7224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [sys.catalog]: This master's current role is: FOLLOWER
W20260618 23:32:00.091344 7235 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:32:00.091425 7235 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:32:00.094405 7151 master_runner.cc:428] Detected that this master 34f7483731cd4088aad3cbc7663639fe is joining an existing cluster
I20260618 23:32:00.094458 7151 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:40637
I20260618 23:32:00.103351 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.096190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56496 (local address 127.0.98.60:40637)
0618 23:32:00.096396 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.096399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.096412 (+ 13us) server_negotiation.cc:407] Connection header received
0618 23:32:00.096443 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.096444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.096456 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.096543 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:00.096973 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.097778 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.098677 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.098993 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.099078 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.100523 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:00.100525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:00.100525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:00.100532 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:00.102220 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.102612 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.102613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.102613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.102658 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.102841 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.102842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.102842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.102967 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:00.103008 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.103182 (+ 174us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.103257 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260618 23:32:00.103713 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.095981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40637 (local address 127.0.0.1:56496)
0618 23:32:00.096277 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.096295 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.096337 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.096606 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.096607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.096776 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:00.096863 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.096865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.097937 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.097938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.098579 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.098581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.099095 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.099096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.099146 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.099714 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:00.099719 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:00.100428 (+ 709us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:00.102305 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.102306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.102307 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.102537 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.102730 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.102731 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.102731 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.102782 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.103047 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:00.103047 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:00.103092 (+ 45us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.103484 (+ 392us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.103610 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":93,"threads_started":1}
I20260618 23:32:00.108120 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.096018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48296 (local address 127.0.98.62:43955)
0618 23:32:00.096071 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.096074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.099816 (+ 3742us) server_negotiation.cc:407] Connection header received
0618 23:32:00.099924 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.099926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.099939 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.099972 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:00.100597 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.101933 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.103895 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.104145 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.104187 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.105252 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:00.105254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:00.105254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:00.105261 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:00.106761 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.107210 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.107212 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.107212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.107289 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.107516 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.107517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.107518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.107694 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:00.107773 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.107946 (+ 173us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.108023 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260618 23:32:00.108134 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.096134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.0.1:48296)
0618 23:32:00.099744 (+ 3610us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.099755 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.099847 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.100038 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.100039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.100428 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:00.100509 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.100511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.102040 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.102041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.103700 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.103703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.104263 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.104264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.104327 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.104754 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:00.104759 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:00.105147 (+ 388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:00.106873 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.106874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.106875 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.107114 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.107375 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.107375 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.107376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.107440 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.107764 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:00.107765 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:00.107821 (+ 56us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.107909 (+ 88us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.108024 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3577,"thread_start_us":80,"threads_started":1}
I20260618 23:32:00.108232 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.096620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35436 (local address 127.0.98.61:36757)
0618 23:32:00.096668 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.096670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.099802 (+ 3132us) server_negotiation.cc:407] Connection header received
0618 23:32:00.099920 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.099921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.099935 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.099968 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:00.100639 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.101412 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.102651 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.102954 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.102997 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.104666 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:00.104668 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:00.104669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:00.104677 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:00.106596 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.107028 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.107030 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.107030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.107105 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.107620 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.107622 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.107622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.107778 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:00.107824 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.108028 (+ 204us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.108120 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260618 23:32:00.108970 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.096071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36757 (local address 127.0.0.1:35436)
0618 23:32:00.099707 (+ 3636us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.099723 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.099837 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.100045 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.100047 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.100450 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:00.100533 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.100536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.101513 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.101514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.102516 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.102519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.103088 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.103089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.103151 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.104061 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:00.104067 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:00.104531 (+ 464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:00.106704 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.106705 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.106705 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.106931 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.107382 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.107383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.107383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.107474 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.107874 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:00.107875 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:00.107921 (+ 46us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.108712 (+ 791us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.108847 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3609,"thread_start_us":106,"threads_started":1}
Jun 18 23:32:00 dist-test-slave-6224 krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825518, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260618 23:32:00.117988 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 6561
I20260618 23:32:00.125411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:32993
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43955
--webserver_port=36425
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43955,127.0.98.61:36757
--tserver_master_addrs=127.0.98.62:43955,127.0.98.61:36757,127.0.98.60:40637 with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:32:00.134505 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.110640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56504 (local address 127.0.98.60:40637)
0618 23:32:00.110694 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.110696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.110703 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:32:00.110731 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.110731 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.110743 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.110804 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:00.111097 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.111619 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.112298 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.112515 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.112591 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.114436 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:00.114438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:00.114438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:00.114446 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:00.133216 (+ 18770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.133651 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.133652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.133653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.133710 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.133949 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.133950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.133950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.134075 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:00.134152 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.134308 (+ 156us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.134402 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260618 23:32:00.134559 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.110451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40637 (local address 127.0.0.1:56504)
0618 23:32:00.110522 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.110534 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.110583 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.110820 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.110821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.110979 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:00.111015 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.111017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.111681 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.111682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.112197 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.112199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.112588 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.112589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.112652 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.113126 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:00.113130 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:00.114346 (+ 1216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:00.133346 (+ 19000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.133348 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.133349 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.133555 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.133780 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.133781 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.133781 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.133846 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.134145 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:00.134145 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:00.134197 (+ 52us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.134284 (+ 87us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.134406 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260618 23:32:00.135263 6498 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:40637: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "34f7483731cd4088aad3cbc7663639fe"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 40637
}
attrs {
promote: true
}
}
dest_uuid: "2270c132876245d0b99153c4cc9f43db"
cas_config_opid_index: 5
I20260618 23:32:00.135550 6498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: true } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: true } }
I20260618 23:32:00.136183 7058 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 34f7483731cd4088aad3cbc7663639fe (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: true } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: true } } }
I20260618 23:32:00.137033 7058 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:40637
I20260618 23:32:00.137123 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2270c132876245d0b99153c4cc9f43db" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: true } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: true } } }
I20260618 23:32:00.137204 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [sys.catalog]: This master's current role is: LEADER
I20260618 23:32:00.137382 7151 master.cc:537] Master@127.0.98.60:40637 shutting down...
W20260618 23:32:00.138213 6483 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db -> Peer 34f7483731cd4088aad3cbc7663639fe (127.0.98.60:40637): Couldn't send request to peer 34f7483731cd4088aad3cbc7663639fe. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260618 23:32:00.141242 7151 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20260618 23:32:00.141341 7151 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20260618 23:32:00.141393 7151 tablet_replica.cc:330] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe: stopping tablet replica
I20260618 23:32:00.142025 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.137421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36757 (local address 127.0.0.1:35438)
0618 23:32:00.137895 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.137907 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.137977 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.138206 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.138207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.138209 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:00.138255 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.138257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.139235 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.139236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.140821 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.140824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.141748 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.141751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.141806 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.141819 (+ 13us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.141910 (+ 91us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.141956 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457}
I20260618 23:32:00.142025 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.137533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35438 (local address 127.0.98.61:36757)
0618 23:32:00.137653 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.137656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.137945 (+ 289us) server_negotiation.cc:407] Connection header received
0618 23:32:00.138053 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.138054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.138078 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.138121 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:00.138335 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.139127 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.140940 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.141562 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.141615 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.141683 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.141919 (+ 236us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.141956 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260618 23:32:00.142380 7107 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [term 1 LEARNER]: Refusing update from remote peer 2270c132876245d0b99153c4cc9f43db: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260618 23:32:00.142730 7245 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260618 23:32:00.143347 7107 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 34f7483731cd4088aad3cbc7663639fe (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: true } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: true } } }
I20260618 23:32:00.145000 7137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2270c132876245d0b99153c4cc9f43db. Latest consensus state: current_term: 1 leader_uuid: "2270c132876245d0b99153c4cc9f43db" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: true } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: true } } }
I20260618 23:32:00.145109 7137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [sys.catalog]: This master's current role is: LEARNER
I20260618 23:32:00.145150 7137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2270c132876245d0b99153c4cc9f43db" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: true } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: true } } }
I20260618 23:32:00.145210 7137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [sys.catalog]: This master's current role is: LEARNER
I20260618 23:32:00.147099 7059 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db: attempting to promote NON_VOTER 22b5aed081f942d599a7192cba8c03a8 to VOTER
I20260618 23:32:00.147325 7059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: true } }
I20260618 23:32:00.147914 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.147800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40637 (local address 127.0.0.1:56510)
0618 23:32:00.147834 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.147881 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260618 23:32:00.147969 7107 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [term 1 LEARNER]: Refusing update from remote peer 2270c132876245d0b99153c4cc9f43db: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260618 23:32:00.148137 7058 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260618 23:32:00.148341 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.148239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40637 (local address 127.0.0.1:56526)
0618 23:32:00.148293 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.148316 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260618 23:32:00.148459 6483 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db -> Peer 34f7483731cd4088aad3cbc7663639fe (127.0.98.60:40637): Couldn't send request to peer 34f7483731cd4088aad3cbc7663639fe. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40637: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260618 23:32:00.148730 7058 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 22b5aed081f942d599a7192cba8c03a8 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: true } } }
I20260618 23:32:00.148901 7107 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 22b5aed081f942d599a7192cba8c03a8 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: true } } }
I20260618 23:32:00.149793 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2270c132876245d0b99153c4cc9f43db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: true } } }
I20260618 23:32:00.149887 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [sys.catalog]: This master's current role is: LEADER
I20260618 23:32:00.149962 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2270c132876245d0b99153c4cc9f43db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: true } } }
I20260618 23:32:00.150012 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [sys.catalog]: This master's current role is: LEADER
I20260618 23:32:00.149968 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2270c132876245d0b99153c4cc9f43db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: true } } }
I20260618 23:32:00.150043 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:32:00.158097 7151 master.cc:552] Master@127.0.98.60:40637 shutdown complete.
I20260618 23:32:00.158164 7151 master_runner.cc:305] Clearing existing system tablet
I20260618 23:32:00.158304 7151 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:00.158366 7151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:00.159024 7151 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260618 23:32:00.159665 7251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:00.159850 7151 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:00.159925 7151 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260618 23:32:00.159961 7151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/wal
uuid: "34f7483731cd4088aad3cbc7663639fe"
format_stamp: "Formatted at 2026-06-18 23:32:00 on dist-test-slave-6224"
server_key: "79d441fce6bf5b6f73e35bb8723f71e2"
server_key_iv: "8d32f6451e37d80a1aaecb45a523d087"
server_key_version: "encryptionkey@0"
I20260618 23:32:00.160708 7151 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:32:00.161984 7151 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260618 23:32:00.162067 7151 log.cc:1196] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260618 23:32:00.162256 7151 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe: Deleting consensus metadata
I20260618 23:32:00.162395 7151 master_runner.cc:315] Copying system tablet from 127.0.98.62:43955
I20260618 23:32:00.163044 7151 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:43955
I20260618 23:32:00.169965 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.163392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.0.1:48310)
0618 23:32:00.163584 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.163596 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.163655 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.163782 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.163783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.163978 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:00.164043 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.164044 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.165023 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.165024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.165625 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.165626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.166134 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.166136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.166208 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.166728 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:00.166733 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:00.167043 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:00.168836 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.168837 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.168837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.169014 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.169223 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.169224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.169225 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.169303 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.169582 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:00.169582 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:00.169627 (+ 45us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.169726 (+ 99us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.169832 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":101,"threads_started":1}
I20260618 23:32:00.170010 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.163572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48310 (local address 127.0.98.62:43955)
0618 23:32:00.163621 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.163623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.163638 (+ 15us) server_negotiation.cc:407] Connection header received
0618 23:32:00.163711 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.163712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.163722 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.163768 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:00.164133 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.164932 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.165830 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.166047 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.166133 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.167154 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:00.167156 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:00.167156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:00.167163 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:00.168726 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.169092 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.169093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.169093 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.169149 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.169396 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.169397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.169397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.169513 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:00.169591 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.169754 (+ 163us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.169830 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:32:00.170606 6519 tablet_copy_service.cc:140] P 2270c132876245d0b99153c4cc9f43db: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 34f7483731cd4088aad3cbc7663639fe ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48310)
I20260618 23:32:00.170665 6519 tablet_copy_service.cc:161] P 2270c132876245d0b99153c4cc9f43db: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 34f7483731cd4088aad3cbc7663639fe at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48310: session id = 34f7483731cd4088aad3cbc7663639fe-00000000000000000000000000000000
I20260618 23:32:00.171201 6519 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db: Tablet Copy: opened 0 blocks and 1 log segments
I20260618 23:32:00.171751 7151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260618 23:32:00.172813 7151 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe: tablet copy: Starting download of 0 data blocks...
I20260618 23:32:00.172968 7151 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe: tablet copy: Starting download of 1 WAL segments...
I20260618 23:32:00.173630 7151 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260618 23:32:00.174352 6519 tablet_copy_service.cc:342] P 2270c132876245d0b99153c4cc9f43db: Request end of tablet copy session 34f7483731cd4088aad3cbc7663639fe-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48310
I20260618 23:32:00.174402 6519 tablet_copy_service.cc:433] P 2270c132876245d0b99153c4cc9f43db: ending tablet copy session 34f7483731cd4088aad3cbc7663639fe-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 34f7483731cd4088aad3cbc7663639fe
I20260618 23:32:00.176779 7151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:00.178028 7258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:00.178064 7261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:00.179773 7151 server_base.cc:1034] running on GCE node
W20260618 23:32:00.184657 7259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:00.184862 7151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:00.185072 7151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:00.186240 7151 hybrid_clock.cc:648] HybridClock initialized: now 1781825520186234 us; error 31 us; skew 500 ppm
Jun 18 23:32:00 dist-test-slave-6224 krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825520, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:32:00.187731 7151 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260618 23:32:00.188019 7151 webserver.cc:460] Webserver started at http://127.0.98.60:41297/ using document root <none> and password file <none>
I20260618 23:32:00.188139 7151 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:00.188175 7151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:00.188805 7151 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260618 23:32:00.189355 7268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:00.189596 7151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260618 23:32:00.189651 7151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/wal
uuid: "34f7483731cd4088aad3cbc7663639fe"
format_stamp: "Formatted at 2026-06-18 23:32:00 on dist-test-slave-6224"
server_key: "79d441fce6bf5b6f73e35bb8723f71e2"
server_key_iv: "8d32f6451e37d80a1aaecb45a523d087"
server_key_version: "encryptionkey@0"
I20260618 23:32:00.189713 7151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:00.205976 7151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:00.207753 7151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:00.210868 7151 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40637
I20260618 23:32:00.211265 7151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/master-2/data/info.pb
I20260618 23:32:00.211280 7320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40637 every 8 connection(s)
I20260618 23:32:00.212100 7321 sys_catalog.cc:263] Verifying existing consensus state
I20260618 23:32:00.212533 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe: Bootstrap starting.
I20260618 23:32:00.214879 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:32:00.215127 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe: Bootstrap complete.
I20260618 23:32:00.215355 7321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: true } }
I20260618 23:32:00.215451 7321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 34f7483731cd4088aad3cbc7663639fe, State: Initialized, Role: LEARNER
I20260618 23:32:00.215498 7321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: true } }
I20260618 23:32:00.215706 7321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:32:00.215698 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: true } } }
I20260618 23:32:00.215916 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [sys.catalog]: This master's current role is: LEARNER
I20260618 23:32:00.217240 7335 catalog_manager.cc:1259] Loaded cluster ID: b45c6bc9c18d4dc7bb5d96301b19a73c
I20260618 23:32:00.217329 7335 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe: loading cluster ID for follower catalog manager: success
I20260618 23:32:00.218461 7335 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe: acquiring CA information for follower catalog manager: success
I20260618 23:32:00.218813 7335 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260618 23:32:00.260008 7244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:00.260396 7244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:00.260440 7244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:00.262447 7244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:00.262537 7244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:00.262589 7244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:32:00.265074 7244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:32993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36425
--tserver_master_addrs=127.0.98.62:43955,127.0.98.61:36757,127.0.98.60:40637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:00.265413 7244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:00.265753 7244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:00.271903 7341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:00.271926 7343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:00.272235 7244 server_base.cc:1034] running on GCE node
W20260618 23:32:00.271896 7340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:00.272634 7244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:00.272974 7244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:00.274178 7244 hybrid_clock.cc:648] HybridClock initialized: now 1781825520274140 us; error 53 us; skew 500 ppm
Jun 18 23:32:00 dist-test-slave-6224 krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825520, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:32:00.277504 7244 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260618 23:32:00.277995 7244 webserver.cc:460] Webserver started at http://127.0.98.1:36425/ using document root <none> and password file <none>
I20260618 23:32:00.278257 7244 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:00.278367 7244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:00.279992 7244 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:00.280750 7350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:00.281002 7244 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:00.281116 7244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "e2015d1fccc3456eae43d83da8ba816a"
format_stamp: "Formatted at 2026-06-18 23:31:59 on dist-test-slave-6224"
server_key: "d30059e8c9bf564c1ed1f5f6af447ac6"
server_key_iv: "96160827797105a0bc88ad44d9aaceaa"
server_key_version: "encryptionkey@0"
I20260618 23:32:00.281524 7244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:00.296789 7244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:00.299119 7244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:00.299279 7244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:00.299522 7244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:32:00.299870 7244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:32:00.299926 7244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:00.299986 7244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:32:00.300032 7244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:00.307132 7244 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32993
I20260618 23:32:00.307260 7463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32993 every 8 connection(s)
I20260618 23:32:00.307648 7244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:32:00.312594 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 7244
I20260618 23:32:00.312731 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 6695
Jun 18 23:32:00 dist-test-slave-6224 krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825520, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 18 23:32:00 dist-test-slave-6224 krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825520, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 18 23:32:00 dist-test-slave-6224 krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825520, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260618 23:32:00.321395 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.308894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46275 (local address 127.0.98.61:36757)
0618 23:32:00.308985 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.308988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.309282 (+ 294us) server_negotiation.cc:407] Connection header received
0618 23:32:00.309917 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.309919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.309931 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.309964 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:00.311012 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.311826 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.315278 (+ 3452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.315606 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.315699 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.317901 (+ 2202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:00.317903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:00.317904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:00.317912 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:00.319831 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.320284 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.320286 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.320286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.320343 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.320565 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.320566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.320566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.320727 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:00.320767 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.321190 (+ 423us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.321282 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 23:32:00.321395 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.308662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33841 (local address 127.0.98.62:43955)
0618 23:32:00.308732 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.308734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.309306 (+ 572us) server_negotiation.cc:407] Connection header received
0618 23:32:00.309895 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.309897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.309909 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.309971 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:00.311006 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.311665 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.313374 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.313570 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.313670 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.317334 (+ 3664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:00.317335 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:00.317336 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:00.317345 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:00.319504 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.320241 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.320243 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.320243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.320308 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.320565 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.320566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.320566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.320717 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:00.320766 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.321179 (+ 413us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.321282 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260618 23:32:00.321456 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.308871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54749 (local address 127.0.98.60:40637)
0618 23:32:00.309046 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.309049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.309269 (+ 220us) server_negotiation.cc:407] Connection header received
0618 23:32:00.309911 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.309912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.309928 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.309969 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:00.311012 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.312049 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.313010 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.313348 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.313402 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.317294 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:00.317296 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:00.317296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:00.317304 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:00.319430 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.320202 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.320203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.320204 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.320263 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.320558 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.320559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.320559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.320712 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:00.320759 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.321186 (+ 427us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.321292 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":68,"threads_started":1}
I20260618 23:32:00.322149 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.308543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40637 (local address 127.0.98.1:54749)
0618 23:32:00.309148 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.309166 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.309776 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.310079 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.310084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.310653 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:00.310904 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.310911 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.312135 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.312135 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.312914 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.312917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.313492 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.313493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.313651 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.314247 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:00.314258 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:00.317182 (+ 2924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:00.319585 (+ 2403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.319591 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.319594 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.320112 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.320333 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.320333 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.320334 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.320484 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.320806 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:00.320812 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:00.321005 (+ 193us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.321577 (+ 572us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.321822 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"mutex_wait_us":32,"spinlock_wait_cycles":8960,"thread_start_us":252,"threads_started":1}
I20260618 23:32:00.322153 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.308543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.1:33841)
0618 23:32:00.309132 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.309165 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.309782 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.310389 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.310390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.310657 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:00.310902 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.310908 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.311799 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.311800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.312744 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.312746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.313672 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.313673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.313756 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.314415 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:00.314421 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:00.317214 (+ 2793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:00.319662 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.319664 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.319664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.320144 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.320386 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.320387 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.320387 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.320488 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.320816 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:00.320816 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:00.321004 (+ 188us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.321568 (+ 564us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.321950 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":189,"thread_start_us":400,"threads_started":1}
I20260618 23:32:00.322161 7469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.308544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36757 (local address 127.0.98.1:46275)
0618 23:32:00.309115 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.309166 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.309776 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.310079 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.310084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.310628 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:00.310904 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.310912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.311927 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.311928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.315187 (+ 3259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.315191 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.315701 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.315702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.315778 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.316261 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:00.316266 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:00.317805 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:00.319925 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.319926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.319927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.320201 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.320412 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.320413 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.320413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.320488 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.320814 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:00.320815 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:00.321004 (+ 189us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.321774 (+ 770us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.322010 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":111,"thread_start_us":124,"threads_started":1}
I20260618 23:32:00.322893 7464 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40637
I20260618 23:32:00.322942 7465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43955
I20260618 23:32:00.323014 7465 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:00.323012 7464 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:00.323252 7465 heartbeater.cc:511] Master 127.0.98.62:43955 requested a full tablet report, sending...
I20260618 23:32:00.323251 7464 heartbeater.cc:511] Master 127.0.98.60:40637 requested a full tablet report, sending...
I20260618 23:32:00.323851 6498 ts_manager.cc:194] Re-registered known tserver with Master: e2015d1fccc3456eae43d83da8ba816a (127.0.98.1:32993)
I20260618 23:32:00.324023 7285 ts_manager.cc:194] Registered new tserver with Master: e2015d1fccc3456eae43d83da8ba816a (127.0.98.1:32993)
I20260618 23:32:00.324183 7466 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36757
I20260618 23:32:00.324229 7466 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:00.324290 7466 heartbeater.cc:511] Master 127.0.98.61:36757 requested a full tablet report, sending...
I20260618 23:32:00.324390 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33841
I20260618 23:32:00.324700 7097 ts_manager.cc:194] Registered new tserver with Master: e2015d1fccc3456eae43d83da8ba816a (127.0.98.1:32993)
I20260618 23:32:00.325237 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35403
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43955
--webserver_port=39821
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43955,127.0.98.61:36757
--tserver_master_addrs=127.0.98.62:43955,127.0.98.61:36757,127.0.98.60:40637 with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:32:00.413308 7472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:00.413509 7472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:00.413533 7472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:00.414666 7472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:00.414713 7472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:00.414740 7472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:32:00.416363 7472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39821
--tserver_master_addrs=127.0.98.62:43955,127.0.98.61:36757,127.0.98.60:40637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:00.416599 7472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:00.416916 7472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:00.422374 7477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:00.422497 7472 server_base.cc:1034] running on GCE node
W20260618 23:32:00.422394 7480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:00.422384 7478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:00.422869 7472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:00.423161 7472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:00.424331 7472 hybrid_clock.cc:648] HybridClock initialized: now 1781825520424327 us; error 40 us; skew 500 ppm
Jun 18 23:32:00 dist-test-slave-6224 krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825520, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:32:00.427449 7472 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260618 23:32:00.427963 7472 webserver.cc:460] Webserver started at http://127.0.98.2:39821/ using document root <none> and password file <none>
I20260618 23:32:00.428215 7472 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:00.428301 7472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:00.429941 7472 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:32:00.430738 7487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:00.431102 7472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:32:00.431216 7472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "8232a14a2ef3499ea04b5f76d538cf83"
format_stamp: "Formatted at 2026-06-18 23:31:59 on dist-test-slave-6224"
server_key: "8476eb42989d9b345fe25a3d61a966e7"
server_key_iv: "43b1880f705e67ef3339d194486d79db"
server_key_version: "encryptionkey@0"
I20260618 23:32:00.431641 7472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:00.456044 7472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:00.458551 7472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:00.458719 7472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:00.458966 7472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:32:00.459307 7472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:32:00.459369 7472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:00.459414 7472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:32:00.459443 7472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:00.466780 7472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35403
I20260618 23:32:00.466833 7600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35403 every 8 connection(s)
I20260618 23:32:00.467192 7472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:32:00.471468 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 7472
I20260618 23:32:00.471683 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 6829
Jun 18 23:32:00 dist-test-slave-6224 krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825520, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 18 23:32:00 dist-test-slave-6224 krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825520, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 18 23:32:00 dist-test-slave-6224 krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825520, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260618 23:32:00.481406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37473
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43955
--webserver_port=43525
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:43955,127.0.98.61:36757
--tserver_master_addrs=127.0.98.62:43955,127.0.98.61:36757,127.0.98.60:40637 with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:32:00.482704 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.467945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40701 (local address 127.0.98.61:36757)
0618 23:32:00.467997 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.467999 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.468817 (+ 818us) server_negotiation.cc:407] Connection header received
0618 23:32:00.469929 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.469931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.469946 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.470000 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:00.471146 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.471979 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.473868 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.474164 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.474206 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.478969 (+ 4763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:00.478970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:00.478970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:00.478978 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:00.480824 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.481558 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.481560 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.481560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.481620 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.481920 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.481921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.481922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.482085 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:00.482130 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.482524 (+ 394us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.482608 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:32:00.482841 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.468250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41661 (local address 127.0.98.60:40637)
0618 23:32:00.468298 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.468300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.468513 (+ 213us) server_negotiation.cc:407] Connection header received
0618 23:32:00.469928 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.469930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.469945 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.470000 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:00.471136 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.471976 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.473879 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.474238 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.474284 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.478596 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:00.478598 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:00.478598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:00.478610 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:00.480837 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.481584 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.481586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.481586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.481656 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.481904 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.481905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.481905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.482081 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:00.482129 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.482529 (+ 400us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.482656 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260618 23:32:00.483174 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.467969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36757 (local address 127.0.98.2:40701)
0618 23:32:00.468724 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.468735 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.469723 (+ 988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.470076 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.470079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.470678 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:00.470998 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.471007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.472126 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.472127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.473758 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.473761 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.474287 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.474287 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.474435 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.475280 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:00.475292 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:00.478870 (+ 3578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:00.480929 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.480936 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.480939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.481470 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.481686 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.481687 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.481687 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.481824 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.482177 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:00.482183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:00.482361 (+ 178us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.482798 (+ 437us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.482904 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"mutex_wait_us":792,"thread_start_us":73,"threads_started":1}
I20260618 23:32:00.483599 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.468499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57253 (local address 127.0.98.62:43955)
0618 23:32:00.468548 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.468550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.468568 (+ 18us) server_negotiation.cc:407] Connection header received
0618 23:32:00.469837 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.469886 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.469902 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.469964 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:00.471105 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.472017 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.473875 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.474164 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.474220 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.478619 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:00.478620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:00.478621 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:00.478630 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:00.481314 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.481965 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.481966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.481967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.482015 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.482226 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.482227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.482227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.482359 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:00.482420 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.483173 (+ 753us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.483475 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260618 23:32:00.483532 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.467970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40637 (local address 127.0.98.2:41661)
0618 23:32:00.468264 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.468346 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.469699 (+ 1353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.470076 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.470079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.470664 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:00.470998 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.471007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.472130 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.472131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.473777 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.473779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.474377 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.474378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.474459 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.475280 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:00.475294 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:00.478442 (+ 3148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:00.480938 (+ 2496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.480940 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.480941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.481469 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.481750 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.481751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.481751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.481825 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.482176 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:00.482183 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:00.482364 (+ 181us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.483077 (+ 713us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.483251 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":79,"thread_start_us":97,"threads_started":1}
I20260618 23:32:00.483731 7603 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36757
I20260618 23:32:00.484019 7603 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:00.484228 7603 heartbeater.cc:511] Master 127.0.98.61:36757 requested a full tablet report, sending...
I20260618 23:32:00.484261 7601 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40637
I20260618 23:32:00.484318 7601 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:00.484426 7601 heartbeater.cc:511] Master 127.0.98.60:40637 requested a full tablet report, sending...
I20260618 23:32:00.484542 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.467973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.2:57253)
0618 23:32:00.468353 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.468363 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.469700 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.470073 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.470077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.470655 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:00.470997 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.471004 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.472126 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.472127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.473785 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.473787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.474280 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.474281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.474435 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.475381 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:00.475387 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:00.478522 (+ 3135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:00.481418 (+ 2896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.481419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.481419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.481890 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.482096 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.482097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.482097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.482159 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.482446 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:00.482447 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:00.482491 (+ 44us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.482563 (+ 72us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.482743 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"mutex_wait_us":251,"thread_start_us":86,"threads_started":1}
I20260618 23:32:00.484762 7097 ts_manager.cc:194] Registered new tserver with Master: 8232a14a2ef3499ea04b5f76d538cf83 (127.0.98.2:35403)
I20260618 23:32:00.484835 7285 ts_manager.cc:194] Registered new tserver with Master: 8232a14a2ef3499ea04b5f76d538cf83 (127.0.98.2:35403)
I20260618 23:32:00.485081 7602 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43955
I20260618 23:32:00.485124 7602 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:00.485194 7602 heartbeater.cc:511] Master 127.0.98.62:43955 requested a full tablet report, sending...
I20260618 23:32:00.485481 6498 ts_manager.cc:194] Re-registered known tserver with Master: 8232a14a2ef3499ea04b5f76d538cf83 (127.0.98.2:35403)
I20260618 23:32:00.485879 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57253
W20260618 23:32:00.581976 7608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:00.582279 7608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:00.582320 7608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:00.583868 7608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:00.583978 7608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:00.584030 7608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:32:00.586141 7608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37473
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43525
--tserver_master_addrs=127.0.98.62:43955,127.0.98.61:36757,127.0.98.60:40637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:00.586483 7608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:00.586786 7608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:00.592690 7613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:00.592681 7614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:00.592721 7608 server_base.cc:1034] running on GCE node
W20260618 23:32:00.592671 7616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:00.593377 7608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:00.593700 7608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:00.594864 7608 hybrid_clock.cc:648] HybridClock initialized: now 1781825520594845 us; error 36 us; skew 500 ppm
Jun 18 23:32:00 dist-test-slave-6224 krb5kdc[6454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825520, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:32:00.597774 7608 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260618 23:32:00.598295 7608 webserver.cc:460] Webserver started at http://127.0.98.3:43525/ using document root <none> and password file <none>
I20260618 23:32:00.598601 7608 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:00.598687 7608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:00.600400 7608 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260618 23:32:00.601236 7623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:00.601550 7608 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:32:00.601660 7608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5"
format_stamp: "Formatted at 2026-06-18 23:31:59 on dist-test-slave-6224"
server_key: "4504b0eb8b92c9578c973b0bd580d69f"
server_key_iv: "3f288b438553d5a382e7e9711df36ac3"
server_key_version: "encryptionkey@0"
I20260618 23:32:00.602092 7608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:00.619100 7608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:00.622260 7608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:00.622398 7608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:00.622715 7608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:32:00.623140 7608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:32:00.623185 7608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:00.623210 7608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:32:00.623224 7608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:00.630137 7608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37473
I20260618 23:32:00.630182 7736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37473 every 8 connection(s)
I20260618 23:32:00.630573 7608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
Jun 18 23:32:00 dist-test-slave-6224 krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825520, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260618 23:32:00.639374 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 7608
Jun 18 23:32:00 dist-test-slave-6224 krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825520, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 18 23:32:00 dist-test-slave-6224 krb5kdc[6454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825520, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260618 23:32:00.644594 7136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.631704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35799 (local address 127.0.98.61:36757)
0618 23:32:00.631760 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.631763 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.632052 (+ 289us) server_negotiation.cc:407] Connection header received
0618 23:32:00.633229 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.633230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.633245 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.633308 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:00.634136 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.634801 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.636335 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.636627 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.636712 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.640994 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:00.640996 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:00.640997 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:00.641006 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:00.642844 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.643372 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.643373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.643373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.643432 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.643721 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.643722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.643722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.643839 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:00.643908 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.644408 (+ 500us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.644494 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:32:00.644769 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.631464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40637 (local address 127.0.98.3:45159)
0618 23:32:00.632110 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.632119 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.633118 (+ 999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.633411 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.633411 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.633843 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:00.634035 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.634038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.635106 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.635106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.636250 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.636253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.636846 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.636847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.636903 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.637780 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:00.637785 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:00.639732 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:00.643288 (+ 3556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.643290 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.643290 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.643596 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.643826 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.643827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.643827 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.643894 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.644217 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:00.644218 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:00.644285 (+ 67us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.644358 (+ 73us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.644483 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":159,"thread_start_us":397,"threads_started":1}
I20260618 23:32:00.644774 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.631464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36757 (local address 127.0.98.3:35799)
0618 23:32:00.631827 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.631905 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.633092 (+ 1187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.633399 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.633405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.633792 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:00.634029 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.634033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.634916 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.634917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.636213 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.636216 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.636728 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.636729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.636888 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.637729 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:00.637734 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:00.640877 (+ 3143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:00.642956 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.642965 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.642969 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.643287 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.643496 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.643497 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.643498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.643654 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.643937 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:00.643942 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:00.644121 (+ 179us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.644225 (+ 104us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.644438 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":3,"thread_start_us":481,"threads_started":1}
I20260618 23:32:00.645359 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.631738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45159 (local address 127.0.98.60:40637)
0618 23:32:00.631830 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.631832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.632167 (+ 335us) server_negotiation.cc:407] Connection header received
0618 23:32:00.633234 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.633235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.633248 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.633282 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:00.634135 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.635007 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.636374 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.636699 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.636768 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.639856 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:00.639858 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:00.639859 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:00.639869 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:00.643199 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.643687 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.643688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.643688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.643755 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.643962 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.643963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.643963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.644127 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:00.644190 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.645152 (+ 962us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.645270 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260618 23:32:00.645709 7739 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36757
I20260618 23:32:00.645833 7739 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:00.646052 7739 heartbeater.cc:511] Master 127.0.98.61:36757 requested a full tablet report, sending...
I20260618 23:32:00.646842 7097 ts_manager.cc:194] Registered new tserver with Master: 0a88d6f8b7bf48ffb3dd2bc35d448cd5 (127.0.98.3:37473)
I20260618 23:32:00.647186 7737 heartbeater.cc:347] Connected to a master server at 127.0.98.60:40637
I20260618 23:32:00.647235 7737 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:00.647317 7737 heartbeater.cc:511] Master 127.0.98.60:40637 requested a full tablet report, sending...
I20260618 23:32:00.647539 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.631665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46061 (local address 127.0.98.62:43955)
0618 23:32:00.631791 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.631794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.632165 (+ 371us) server_negotiation.cc:407] Connection header received
0618 23:32:00.633234 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.633235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.633248 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.633282 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:00.634136 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.634944 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.636351 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.636575 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.636609 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.640991 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:00.640992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:00.640992 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:00.641001 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:00.644200 (+ 3199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.646279 (+ 2079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.646281 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.646281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.646359 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.646563 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.646563 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.646564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.646698 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:00.646763 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.647023 (+ 260us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.647118 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260618 23:32:00.647672 7285 ts_manager.cc:194] Registered new tserver with Master: 0a88d6f8b7bf48ffb3dd2bc35d448cd5 (127.0.98.3:37473)
I20260618 23:32:00.647859 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.631463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.3:46061)
0618 23:32:00.632048 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.632062 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.633118 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.633411 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.633411 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.633845 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:00.634035 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.634038 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.635072 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.635073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.636250 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.636253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.636716 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.636717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.636894 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.637644 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:00.637671 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:00.640876 (+ 3205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:00.644290 (+ 3414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.644290 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.644291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.646182 (+ 1891us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.646434 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:00.646434 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:00.646434 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:00.646496 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:00.646799 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:00.646800 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:00.646845 (+ 45us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.647600 (+ 755us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.647752 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":116,"thread_start_us":245,"threads_started":1}
I20260618 23:32:00.648363 7738 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43955
I20260618 23:32:00.648427 7738 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:00.648487 7738 heartbeater.cc:511] Master 127.0.98.62:43955 requested a full tablet report, sending...
I20260618 23:32:00.648691 6498 ts_manager.cc:194] Re-registered known tserver with Master: 0a88d6f8b7bf48ffb3dd2bc35d448cd5 (127.0.98.3:37473)
I20260618 23:32:00.649257 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46061
I20260618 23:32:00.650841 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.641506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35448 (local address 127.0.98.61:36757)
0618 23:32:00.641669 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.641671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.642069 (+ 398us) server_negotiation.cc:407] Connection header received
0618 23:32:00.642137 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.642138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.642153 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.642223 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:00.643045 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.643827 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.644974 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.645276 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.645333 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.646920 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:00.646921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:00.646921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:00.646928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:00.649327 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.649776 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.649777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.649777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.649836 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.650052 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.650053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.650053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.650369 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:00.650473 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.650662 (+ 189us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.650744 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260618 23:32:00.653182 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.641160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56540 (local address 127.0.98.60:40637)
0618 23:32:00.642162 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.642164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.642354 (+ 190us) server_negotiation.cc:407] Connection header received
0618 23:32:00.642379 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.642380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.642395 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.642464 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:00.645458 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.646277 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.647382 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.647716 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.647793 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.649581 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:00.649583 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:00.649583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:00.649592 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:00.651715 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.652106 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.652107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.652108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.652173 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.652462 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.652463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.652464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.652651 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:00.652762 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.652970 (+ 208us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.653079 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":982,"thread_start_us":68,"threads_started":1}
I20260618 23:32:00.656601 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.641784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48314 (local address 127.0.98.62:43955)
0618 23:32:00.641935 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.641937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.648486 (+ 6549us) server_negotiation.cc:407] Connection header received
0618 23:32:00.648512 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.648513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.648528 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.648606 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:00.650237 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.651227 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.651999 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.652344 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.652420 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.653796 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:00.653797 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:00.653797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:00.653804 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:00.655400 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.655830 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.655831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.655832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.655895 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:00.656073 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:00.656073 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:00.656074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:00.656182 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:00.656290 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.656375 (+ 85us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.656451 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260618 23:32:00.658469 6498 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:32:00.660912 6498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:32:00.670905 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.666457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35403 (local address 127.0.0.1:51814)
0618 23:32:00.666825 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.666836 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.666901 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.667107 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.667108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.667111 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:00.667178 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.667180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.668276 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.668277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.669457 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.669460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.670601 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.670602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.670674 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.670680 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.670736 (+ 56us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.670800 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":84,"threads_started":1}
I20260618 23:32:00.670914 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.666557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51814 (local address 127.0.98.2:35403)
0618 23:32:00.666766 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.666772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.666960 (+ 188us) server_negotiation.cc:407] Connection header received
0618 23:32:00.666996 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.666998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.667011 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.667099 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:00.667259 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.668213 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.669609 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.670485 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.670598 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.670723 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.670757 (+ 34us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.670807 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":97,"threads_started":1}
I20260618 23:32:00.671716 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.666236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37473 (local address 127.0.0.1:39814)
0618 23:32:00.666377 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.666393 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.666454 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.667111 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.667113 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.667116 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:00.667215 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.667218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.668766 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.668767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.669966 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.669968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.671411 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.671412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.671481 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.671487 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.671522 (+ 35us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.671621 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260618 23:32:00.671783 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.666712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38390 (local address 127.0.98.1:32993)
0618 23:32:00.666934 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.666939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.666949 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:32:00.666994 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.666998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.667018 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.667113 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:00.667324 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.668658 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.670056 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.671258 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.671336 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.671420 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.671648 (+ 228us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.671692 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":106,"threads_started":1}
I20260618 23:32:00.671850 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.666236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32993 (local address 127.0.0.1:38390)
0618 23:32:00.666400 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.666412 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.666560 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.667196 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.667196 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.667198 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:00.667238 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.667240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.668787 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.668789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.669928 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.669932 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.671432 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.671433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.671505 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.671511 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.671622 (+ 111us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.671681 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":3,"thread_start_us":107,"threads_started":1}
I20260618 23:32:00.672089 7535 tablet_service.cc:1467] Processing CreateTablet for tablet cf840a12daa649a0b4a7eb2d1efefd0c (DEFAULT_TABLE table=test-table [id=901e84eb5b43471eae3b317949589169]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:32:00.672457 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.666548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39814 (local address 127.0.98.3:37473)
0618 23:32:00.666859 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.666866 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.666882 (+ 16us) server_negotiation.cc:407] Connection header received
0618 23:32:00.666938 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.666941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.666965 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.667307 (+ 342us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:00.667370 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.668670 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.670096 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.670894 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.672163 (+ 1269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.672255 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.672282 (+ 27us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.672335 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":126,"threads_started":1}
I20260618 23:32:00.672600 7535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf840a12daa649a0b4a7eb2d1efefd0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:00.672863 7398 tablet_service.cc:1467] Processing CreateTablet for tablet cf840a12daa649a0b4a7eb2d1efefd0c (DEFAULT_TABLE table=test-table [id=901e84eb5b43471eae3b317949589169]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:32:00.673216 7398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf840a12daa649a0b4a7eb2d1efefd0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:00.676179 7765 tablet_bootstrap.cc:492] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83: Bootstrap starting.
I20260618 23:32:00.676419 7671 tablet_service.cc:1467] Processing CreateTablet for tablet cf840a12daa649a0b4a7eb2d1efefd0c (DEFAULT_TABLE table=test-table [id=901e84eb5b43471eae3b317949589169]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:32:00.676792 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf840a12daa649a0b4a7eb2d1efefd0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:00.677001 7765 tablet_bootstrap.cc:654] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:00.677312 7765 log.cc:824] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83: Log is configured to *not* fsync() on all Append() calls
I20260618 23:32:00.678098 7765 tablet_bootstrap.cc:492] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83: No bootstrap required, opened a new log
I20260618 23:32:00.678164 7765 ts_tablet_manager.cc:1397] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:32:00.679455 7768 tablet_bootstrap.cc:492] T cf840a12daa649a0b4a7eb2d1efefd0c P 0a88d6f8b7bf48ffb3dd2bc35d448cd5: Bootstrap starting.
I20260618 23:32:00.679716 7766 tablet_bootstrap.cc:492] T cf840a12daa649a0b4a7eb2d1efefd0c P e2015d1fccc3456eae43d83da8ba816a: Bootstrap starting.
I20260618 23:32:00.680191 7768 tablet_bootstrap.cc:654] T cf840a12daa649a0b4a7eb2d1efefd0c P 0a88d6f8b7bf48ffb3dd2bc35d448cd5: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:00.680285 7766 tablet_bootstrap.cc:654] T cf840a12daa649a0b4a7eb2d1efefd0c P e2015d1fccc3456eae43d83da8ba816a: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:00.680536 7768 log.cc:824] T cf840a12daa649a0b4a7eb2d1efefd0c P 0a88d6f8b7bf48ffb3dd2bc35d448cd5: Log is configured to *not* fsync() on all Append() calls
I20260618 23:32:00.680585 7766 log.cc:824] T cf840a12daa649a0b4a7eb2d1efefd0c P e2015d1fccc3456eae43d83da8ba816a: Log is configured to *not* fsync() on all Append() calls
I20260618 23:32:00.680826 7765 raft_consensus.cc:348] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37473 } } peers { permanent_uuid: "e2015d1fccc3456eae43d83da8ba816a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32993 } } peers { permanent_uuid: "8232a14a2ef3499ea04b5f76d538cf83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35403 } }
I20260618 23:32:00.681082 7765 raft_consensus.cc:374] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:00.681139 7765 raft_consensus.cc:729] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8232a14a2ef3499ea04b5f76d538cf83, State: Initialized, Role: FOLLOWER
I20260618 23:32:00.681293 7765 consensus_queue.cc:260] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37473 } } peers { permanent_uuid: "e2015d1fccc3456eae43d83da8ba816a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32993 } } peers { permanent_uuid: "8232a14a2ef3499ea04b5f76d538cf83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35403 } }
I20260618 23:32:00.681643 7766 tablet_bootstrap.cc:492] T cf840a12daa649a0b4a7eb2d1efefd0c P e2015d1fccc3456eae43d83da8ba816a: No bootstrap required, opened a new log
I20260618 23:32:00.681614 7765 ts_tablet_manager.cc:1428] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260618 23:32:00.681733 7766 ts_tablet_manager.cc:1397] T cf840a12daa649a0b4a7eb2d1efefd0c P e2015d1fccc3456eae43d83da8ba816a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:32:00.681886 7602 heartbeater.cc:503] Master 127.0.98.62:43955 was elected leader, sending a full tablet report...
I20260618 23:32:00.682201 7768 tablet_bootstrap.cc:492] T cf840a12daa649a0b4a7eb2d1efefd0c P 0a88d6f8b7bf48ffb3dd2bc35d448cd5: No bootstrap required, opened a new log
I20260618 23:32:00.682284 7768 ts_tablet_manager.cc:1397] T cf840a12daa649a0b4a7eb2d1efefd0c P 0a88d6f8b7bf48ffb3dd2bc35d448cd5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 23:32:00.684258 7766 raft_consensus.cc:348] T cf840a12daa649a0b4a7eb2d1efefd0c P e2015d1fccc3456eae43d83da8ba816a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37473 } } peers { permanent_uuid: "e2015d1fccc3456eae43d83da8ba816a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32993 } } peers { permanent_uuid: "8232a14a2ef3499ea04b5f76d538cf83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35403 } }
I20260618 23:32:00.684455 7766 raft_consensus.cc:374] T cf840a12daa649a0b4a7eb2d1efefd0c P e2015d1fccc3456eae43d83da8ba816a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:00.684510 7766 raft_consensus.cc:729] T cf840a12daa649a0b4a7eb2d1efefd0c P e2015d1fccc3456eae43d83da8ba816a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2015d1fccc3456eae43d83da8ba816a, State: Initialized, Role: FOLLOWER
I20260618 23:32:00.684656 7766 consensus_queue.cc:260] T cf840a12daa649a0b4a7eb2d1efefd0c P e2015d1fccc3456eae43d83da8ba816a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37473 } } peers { permanent_uuid: "e2015d1fccc3456eae43d83da8ba816a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32993 } } peers { permanent_uuid: "8232a14a2ef3499ea04b5f76d538cf83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35403 } }
I20260618 23:32:00.684964 7766 ts_tablet_manager.cc:1428] T cf840a12daa649a0b4a7eb2d1efefd0c P e2015d1fccc3456eae43d83da8ba816a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:32:00.685091 7465 heartbeater.cc:503] Master 127.0.98.62:43955 was elected leader, sending a full tablet report...
I20260618 23:32:00.686362 7768 raft_consensus.cc:348] T cf840a12daa649a0b4a7eb2d1efefd0c P 0a88d6f8b7bf48ffb3dd2bc35d448cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37473 } } peers { permanent_uuid: "e2015d1fccc3456eae43d83da8ba816a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32993 } } peers { permanent_uuid: "8232a14a2ef3499ea04b5f76d538cf83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35403 } }
I20260618 23:32:00.686582 7768 raft_consensus.cc:374] T cf840a12daa649a0b4a7eb2d1efefd0c P 0a88d6f8b7bf48ffb3dd2bc35d448cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:00.686631 7768 raft_consensus.cc:729] T cf840a12daa649a0b4a7eb2d1efefd0c P 0a88d6f8b7bf48ffb3dd2bc35d448cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a88d6f8b7bf48ffb3dd2bc35d448cd5, State: Initialized, Role: FOLLOWER
I20260618 23:32:00.686770 7768 consensus_queue.cc:260] T cf840a12daa649a0b4a7eb2d1efefd0c P 0a88d6f8b7bf48ffb3dd2bc35d448cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37473 } } peers { permanent_uuid: "e2015d1fccc3456eae43d83da8ba816a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32993 } } peers { permanent_uuid: "8232a14a2ef3499ea04b5f76d538cf83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35403 } }
I20260618 23:32:00.687089 7768 ts_tablet_manager.cc:1428] T cf840a12daa649a0b4a7eb2d1efefd0c P 0a88d6f8b7bf48ffb3dd2bc35d448cd5: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20260618 23:32:00.687130 7738 heartbeater.cc:503] Master 127.0.98.62:43955 was elected leader, sending a full tablet report...
W20260618 23:32:00.718183 7604 tablet.cc:2369] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:32:00.721763 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.718817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56544 (local address 127.0.98.60:40637)
0618 23:32:00.719046 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.719048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.719059 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:32:00.719091 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.719091 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.719116 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.719194 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:00.719383 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.720017 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.720985 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.721414 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.721514 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.721603 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.721633 (+ 30us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.721674 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260618 23:32:00.721763 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.718689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40637 (local address 127.0.0.1:56544)
0618 23:32:00.718775 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.718799 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.718883 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.719216 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.719217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.719219 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:00.719290 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.719292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.720117 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.720118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.720885 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.720886 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.721513 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.721514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.721570 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.721575 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.721613 (+ 38us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.721670 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260618 23:32:00.722355 7295 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260618 23:32:00.723356 7774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2270c132876245d0b99153c4cc9f43db. Latest consensus state: current_term: 1 leader_uuid: "2270c132876245d0b99153c4cc9f43db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: true } } }
I20260618 23:32:00.723711 7774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [sys.catalog]: This master's current role is: LEARNER
W20260618 23:32:00.809600 7467 tablet.cc:2369] T cf840a12daa649a0b4a7eb2d1efefd0c P e2015d1fccc3456eae43d83da8ba816a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:32:00.882246 7740 tablet.cc:2369] T cf840a12daa649a0b4a7eb2d1efefd0c P 0a88d6f8b7bf48ffb3dd2bc35d448cd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:32:00.960572 7771 raft_consensus.cc:482] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:32:00.960772 7771 raft_consensus.cc:504] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37473 } } peers { permanent_uuid: "e2015d1fccc3456eae43d83da8ba816a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32993 } } peers { permanent_uuid: "8232a14a2ef3499ea04b5f76d538cf83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35403 } }
I20260618 23:32:00.961243 7771 leader_election.cc:288] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a88d6f8b7bf48ffb3dd2bc35d448cd5 (127.0.98.3:37473), e2015d1fccc3456eae43d83da8ba816a (127.0.98.1:32993)
I20260618 23:32:00.965963 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.961613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54619 (local address 127.0.98.3:37473)
0618 23:32:00.961676 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.961679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.961826 (+ 147us) server_negotiation.cc:407] Connection header received
0618 23:32:00.961861 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.961863 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.961879 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.961956 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:00.962322 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.963132 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.964872 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.965470 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.965563 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.965578 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.965825 (+ 247us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.965877 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260618 23:32:00.965950 7605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.961550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37473 (local address 127.0.98.2:54619)
0618 23:32:00.961731 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.961743 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.961803 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.962030 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.962031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.962033 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:00.962168 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.962170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.963257 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.963258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.964737 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.964740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.965596 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.965598 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.965686 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.965691 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.965794 (+ 103us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.965837 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":159}
I20260618 23:32:00.966110 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.961640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60351 (local address 127.0.98.1:32993)
0618 23:32:00.961690 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:00.961692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:00.961703 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:32:00.961848 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:00.961850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:00.961868 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:00.961956 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:00.962322 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.963132 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.964871 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.965476 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.965563 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.965580 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:00.965946 (+ 366us) server_negotiation.cc:299] Negotiation successful
0618 23:32:00.966029 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260618 23:32:00.966223 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:00.961528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32993 (local address 127.0.98.2:60351)
0618 23:32:00.961636 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:00.961650 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:00.961702 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:00.962030 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:00.962031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:00.962035 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:00.962168 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.962170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.963257 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.963258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.964737 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:00.964740 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:00.965686 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:00.965688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:00.965838 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:00.965842 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:32:00.965904 (+ 62us) client_negotiation.cc:240] Negotiation successful
0618 23:32:00.965965 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":131}
I20260618 23:32:00.966421 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf840a12daa649a0b4a7eb2d1efefd0c" candidate_uuid: "8232a14a2ef3499ea04b5f76d538cf83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" is_pre_election: true
I20260618 23:32:00.966620 7691 raft_consensus.cc:2454] T cf840a12daa649a0b4a7eb2d1efefd0c P 0a88d6f8b7bf48ffb3dd2bc35d448cd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8232a14a2ef3499ea04b5f76d538cf83 in term 0.
I20260618 23:32:00.966652 7418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf840a12daa649a0b4a7eb2d1efefd0c" candidate_uuid: "8232a14a2ef3499ea04b5f76d538cf83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2015d1fccc3456eae43d83da8ba816a" is_pre_election: true
I20260618 23:32:00.966859 7418 raft_consensus.cc:2454] T cf840a12daa649a0b4a7eb2d1efefd0c P e2015d1fccc3456eae43d83da8ba816a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8232a14a2ef3499ea04b5f76d538cf83 in term 0.
I20260618 23:32:00.966996 7490 leader_election.cc:302] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a88d6f8b7bf48ffb3dd2bc35d448cd5, 8232a14a2ef3499ea04b5f76d538cf83; no voters:
I20260618 23:32:00.967208 7771 raft_consensus.cc:2781] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:32:00.967272 7771 raft_consensus.cc:482] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:32:00.967289 7771 raft_consensus.cc:3037] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:00.968187 7771 raft_consensus.cc:504] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37473 } } peers { permanent_uuid: "e2015d1fccc3456eae43d83da8ba816a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32993 } } peers { permanent_uuid: "8232a14a2ef3499ea04b5f76d538cf83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35403 } }
I20260618 23:32:00.968358 7771 leader_election.cc:288] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83 [CANDIDATE]: Term 1 election: Requested vote from peers 0a88d6f8b7bf48ffb3dd2bc35d448cd5 (127.0.98.3:37473), e2015d1fccc3456eae43d83da8ba816a (127.0.98.1:32993)
I20260618 23:32:00.968562 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf840a12daa649a0b4a7eb2d1efefd0c" candidate_uuid: "8232a14a2ef3499ea04b5f76d538cf83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5"
I20260618 23:32:00.968623 7418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf840a12daa649a0b4a7eb2d1efefd0c" candidate_uuid: "8232a14a2ef3499ea04b5f76d538cf83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2015d1fccc3456eae43d83da8ba816a"
I20260618 23:32:00.968665 7691 raft_consensus.cc:3037] T cf840a12daa649a0b4a7eb2d1efefd0c P 0a88d6f8b7bf48ffb3dd2bc35d448cd5 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:00.968716 7418 raft_consensus.cc:3037] T cf840a12daa649a0b4a7eb2d1efefd0c P e2015d1fccc3456eae43d83da8ba816a [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:00.969566 7418 raft_consensus.cc:2454] T cf840a12daa649a0b4a7eb2d1efefd0c P e2015d1fccc3456eae43d83da8ba816a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8232a14a2ef3499ea04b5f76d538cf83 in term 1.
I20260618 23:32:00.969578 7691 raft_consensus.cc:2454] T cf840a12daa649a0b4a7eb2d1efefd0c P 0a88d6f8b7bf48ffb3dd2bc35d448cd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8232a14a2ef3499ea04b5f76d538cf83 in term 1.
I20260618 23:32:00.969759 7489 leader_election.cc:302] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8232a14a2ef3499ea04b5f76d538cf83, e2015d1fccc3456eae43d83da8ba816a; no voters:
I20260618 23:32:00.969870 7771 raft_consensus.cc:2781] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:32:00.970016 7771 raft_consensus.cc:686] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83 [term 1 LEADER]: Becoming Leader. State: Replica: 8232a14a2ef3499ea04b5f76d538cf83, State: Running, Role: LEADER
I20260618 23:32:00.970142 7771 consensus_queue.cc:237] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37473 } } peers { permanent_uuid: "e2015d1fccc3456eae43d83da8ba816a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32993 } } peers { permanent_uuid: "8232a14a2ef3499ea04b5f76d538cf83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35403 } }
I20260618 23:32:00.971192 6498 catalog_manager.cc:5495] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8232a14a2ef3499ea04b5f76d538cf83 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "8232a14a2ef3499ea04b5f76d538cf83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2015d1fccc3456eae43d83da8ba816a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8232a14a2ef3499ea04b5f76d538cf83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35403 } health_report { overall_health: HEALTHY } } }
I20260618 23:32:00.972110 7755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2270c132876245d0b99153c4cc9f43db" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: true } } }
I20260618 23:32:00.972247 7755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [sys.catalog]: This master's current role is: LEADER
I20260618 23:32:00.973232 7781 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db: attempting to promote NON_VOTER 34f7483731cd4088aad3cbc7663639fe to VOTER
I20260618 23:32:00.973388 7781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: false } }
I20260618 23:32:00.973835 7107 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [term 1 FOLLOWER]: Refusing update from remote peer 2270c132876245d0b99153c4cc9f43db: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260618 23:32:00.974057 7295 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [term 1 LEARNER]: Refusing update from remote peer 2270c132876245d0b99153c4cc9f43db: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260618 23:32:00.974084 7755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260618 23:32:00.974314 7755 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [LEADER]: Connected to new peer: Peer: permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260618 23:32:00.974787 7248 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 34f7483731cd4088aad3cbc7663639fe (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: false } } }
I20260618 23:32:00.974944 7107 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 34f7483731cd4088aad3cbc7663639fe (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: false } } }
I20260618 23:32:00.975890 7248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2270c132876245d0b99153c4cc9f43db" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: false } } }
I20260618 23:32:00.975992 7248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [sys.catalog]: This master's current role is: LEADER
I20260618 23:32:00.976111 7248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2270c132876245d0b99153c4cc9f43db" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: false } } }
I20260618 23:32:00.976177 7248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [sys.catalog]: This master's current role is: LEADER
I20260618 23:32:00.976151 7783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2270c132876245d0b99153c4cc9f43db" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: false } } }
I20260618 23:32:00.976246 7783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:32:00.976461 7755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2270c132876245d0b99153c4cc9f43db" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: false } } }
I20260618 23:32:00.976573 7755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db [sys.catalog]: This master's current role is: LEADER
I20260618 23:32:00.976603 7295 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 34f7483731cd4088aad3cbc7663639fe (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: false } } }
I20260618 23:32:00.977545 7774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2270c132876245d0b99153c4cc9f43db" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: false } } }
I20260618 23:32:00.977638 7774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:32:01.024910 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:01.022178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51820 (local address 127.0.98.2:35403)
0618 23:32:01.022227 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:01.022229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:01.022256 (+ 27us) server_negotiation.cc:407] Connection header received
0618 23:32:01.022349 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:01.022350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:01.022372 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:01.022452 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:32:01.022684 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:01.023160 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:01.024097 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:01.024293 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:01.024363 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:01.024521 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:32:01.024633 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:32:01.024703 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:01.024765 (+ 62us) server_negotiation.cc:299] Negotiation successful
0618 23:32:01.024786 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260618 23:32:01.027045 7418 raft_consensus.cc:1261] T cf840a12daa649a0b4a7eb2d1efefd0c P e2015d1fccc3456eae43d83da8ba816a [term 1 FOLLOWER]: Refusing update from remote peer 8232a14a2ef3499ea04b5f76d538cf83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:32:01.027065 7691 raft_consensus.cc:1261] T cf840a12daa649a0b4a7eb2d1efefd0c P 0a88d6f8b7bf48ffb3dd2bc35d448cd5 [term 1 FOLLOWER]: Refusing update from remote peer 8232a14a2ef3499ea04b5f76d538cf83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:32:01.027463 7779 consensus_queue.cc:1035] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2015d1fccc3456eae43d83da8ba816a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:32:01.027735 7771 consensus_queue.cc:1035] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:32:01.030707 7785 mvcc.cc:205] Tried to move back new op lower bound from 7298357334123368448 to 7298357333894340608. Current Snapshot: MvccSnapshot[applied={T|T < 7298357334123368448}]
I20260618 23:32:01.031822 7786 mvcc.cc:205] Tried to move back new op lower bound from 7298357334123368448 to 7298357333894340608. Current Snapshot: MvccSnapshot[applied={T|T < 7298357334123368448}]
I20260618 23:32:01.032618 7787 mvcc.cc:205] Tried to move back new op lower bound from 7298357334123368448 to 7298357333894340608. Current Snapshot: MvccSnapshot[applied={T|T < 7298357334123368448 or (T in {7298357334123368448})}]
I20260618 23:32:01.035864 6498 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:32:01.035987 6498 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:32:01.037622 6498 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2270c132876245d0b99153c4cc9f43db: Sending DeleteTablet for 3 replicas of tablet cf840a12daa649a0b4a7eb2d1efefd0c
I20260618 23:32:01.038038 7535 tablet_service.cc:1514] Processing DeleteTablet for tablet cf840a12daa649a0b4a7eb2d1efefd0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:32:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51814
I20260618 23:32:01.038043 7398 tablet_service.cc:1514] Processing DeleteTablet for tablet cf840a12daa649a0b4a7eb2d1efefd0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:32:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38390
I20260618 23:32:01.038296 7794 tablet_replica.cc:330] T cf840a12daa649a0b4a7eb2d1efefd0c P e2015d1fccc3456eae43d83da8ba816a: stopping tablet replica
I20260618 23:32:01.038412 7794 raft_consensus.cc:2229] T cf840a12daa649a0b4a7eb2d1efefd0c P e2015d1fccc3456eae43d83da8ba816a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:32:01.038538 7794 raft_consensus.cc:2258] T cf840a12daa649a0b4a7eb2d1efefd0c P e2015d1fccc3456eae43d83da8ba816a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:32:01.038825 7671 tablet_service.cc:1514] Processing DeleteTablet for tablet cf840a12daa649a0b4a7eb2d1efefd0c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:32:01 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39814
I20260618 23:32:01.039568 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 6467
I20260618 23:32:01.040524 7794 ts_tablet_manager.cc:1905] T cf840a12daa649a0b4a7eb2d1efefd0c P e2015d1fccc3456eae43d83da8ba816a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:32:01.042254 7794 ts_tablet_manager.cc:1918] T cf840a12daa649a0b4a7eb2d1efefd0c P e2015d1fccc3456eae43d83da8ba816a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:32:01.042389 7794 log.cc:1196] T cf840a12daa649a0b4a7eb2d1efefd0c P e2015d1fccc3456eae43d83da8ba816a: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-0/wal/wals/cf840a12daa649a0b4a7eb2d1efefd0c
I20260618 23:32:01.042707 7794 ts_tablet_manager.cc:1939] T cf840a12daa649a0b4a7eb2d1efefd0c P e2015d1fccc3456eae43d83da8ba816a: Deleting consensus metadata
W20260618 23:32:01.046684 7353 connection.cc:535] server connection from 127.0.0.1:38390 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260618 23:32:01.039615 7797 tablet_replica.cc:330] T cf840a12daa649a0b4a7eb2d1efefd0c P 0a88d6f8b7bf48ffb3dd2bc35d448cd5: stopping tablet replica
I20260618 23:32:01.047736 7797 raft_consensus.cc:2229] T cf840a12daa649a0b4a7eb2d1efefd0c P 0a88d6f8b7bf48ffb3dd2bc35d448cd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:32:01.047914 7797 raft_consensus.cc:2258] T cf840a12daa649a0b4a7eb2d1efefd0c P 0a88d6f8b7bf48ffb3dd2bc35d448cd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:32:01.048273 7795 tablet_replica.cc:330] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83: stopping tablet replica
I20260618 23:32:01.048393 7795 raft_consensus.cc:2229] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:32:01.048545 7795 raft_consensus.cc:2258] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:32:01.048960 7797 ts_tablet_manager.cc:1905] T cf840a12daa649a0b4a7eb2d1efefd0c P 0a88d6f8b7bf48ffb3dd2bc35d448cd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:32:01.050113 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:01.039073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56554 (local address 127.0.98.60:40637)
0618 23:32:01.039133 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:01.039137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:01.039222 (+ 85us) server_negotiation.cc:407] Connection header received
0618 23:32:01.039331 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:01.039332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:01.039344 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:01.039377 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:01.042792 (+ 3415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:01.043434 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:01.044266 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:01.044476 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:01.044564 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:01.046003 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:01.046005 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:01.046005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:01.046013 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:01.047837 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:01.049047 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:01.049049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:01.049049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:01.049110 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:01.049395 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:01.049396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:01.049396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:01.049541 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:01.049626 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:01.049862 (+ 236us) server_negotiation.cc:299] Negotiation successful
0618 23:32:01.049960 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260618 23:32:01.050654 7795 ts_tablet_manager.cc:1905] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:32:01.052332 7795 ts_tablet_manager.cc:1918] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:32:01.052429 7795 log.cc:1196] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/cf840a12daa649a0b4a7eb2d1efefd0c
I20260618 23:32:01.052724 7795 ts_tablet_manager.cc:1939] T cf840a12daa649a0b4a7eb2d1efefd0c P 8232a14a2ef3499ea04b5f76d538cf83: Deleting consensus metadata
W20260618 23:32:01.053098 7488 connection.cc:441] server connection from 127.0.0.1:51814 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51814 (request call id 1) could send its response
I20260618 23:32:01.053875 7797 ts_tablet_manager.cc:1918] T cf840a12daa649a0b4a7eb2d1efefd0c P 0a88d6f8b7bf48ffb3dd2bc35d448cd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:32:01.053962 7797 log.cc:1196] T cf840a12daa649a0b4a7eb2d1efefd0c P 0a88d6f8b7bf48ffb3dd2bc35d448cd5: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/cf840a12daa649a0b4a7eb2d1efefd0c
I20260618 23:32:01.054227 7797 ts_tablet_manager.cc:1939] T cf840a12daa649a0b4a7eb2d1efefd0c P 0a88d6f8b7bf48ffb3dd2bc35d448cd5: Deleting consensus metadata
W20260618 23:32:01.054559 7625 connection.cc:441] server connection from 127.0.0.1:39814 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39814 (request call id 1) could send its response
I20260618 23:32:01.058853 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:01.048848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56562 (local address 127.0.98.60:40637)
0618 23:32:01.048900 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:01.048902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:01.049298 (+ 396us) server_negotiation.cc:407] Connection header received
0618 23:32:01.049395 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:01.049396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:01.049408 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:01.049440 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:01.050142 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:01.050928 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:01.052829 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:01.053165 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:01.053212 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:01.055146 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:01.055148 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:01.055148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:01.055155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:01.057095 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:01.057663 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:01.057664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:01.057665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:01.057730 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:01.058237 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:01.058238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:01.058239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:01.058397 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:01.058535 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:01.058638 (+ 103us) server_negotiation.cc:299] Negotiation successful
0618 23:32:01.058732 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260618 23:32:01.060979 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:01.051876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35450 (local address 127.0.98.61:36757)
0618 23:32:01.051929 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:01.051931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:01.052272 (+ 341us) server_negotiation.cc:407] Connection header received
0618 23:32:01.052366 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:01.052367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:01.052384 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:01.052413 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:01.053260 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:01.054401 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:01.055445 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:01.055819 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:01.055891 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:01.057468 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:01.057470 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:01.057470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:01.057478 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:01.059366 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:01.059841 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:01.059843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:01.059843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:01.059913 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:01.060289 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:01.060290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:01.060291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:01.060422 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:01.060540 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:01.060735 (+ 195us) server_negotiation.cc:299] Negotiation successful
0618 23:32:01.060841 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:32:02.032409 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.031946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.1:45489)
0618 23:32:02.032194 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.032318 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":135,"threads_started":1}
W20260618 23:32:02.033049 7465 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43955 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
I20260618 23:32:02.033360 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.033024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.3:55955)
0618 23:32:02.033230 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.033306 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":73,"threads_started":1}
I20260618 23:32:02.033412 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.033156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.2:46187)
0618 23:32:02.033302 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.033362 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
W20260618 23:32:02.033581 7602 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43955 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
W20260618 23:32:02.033599 7738 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43955 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
I20260618 23:32:02.538067 7808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2270c132876245d0b99153c4cc9f43db)
I20260618 23:32:02.538223 7808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: false } }
I20260618 23:32:02.538586 7808 leader_election.cc:288] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2270c132876245d0b99153c4cc9f43db (127.0.98.62:43955), 22b5aed081f942d599a7192cba8c03a8 (127.0.98.61:36757)
I20260618 23:32:02.538628 7809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2270c132876245d0b99153c4cc9f43db)
I20260618 23:32:02.539002 7809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: false } }
I20260618 23:32:02.539088 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.538774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.0.1:37430)
0618 23:32:02.538980 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.539039 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":67,"threads_started":1}
W20260618 23:32:02.539278 7272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
I20260618 23:32:02.539819 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.539714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.0.1:37458)
0618 23:32:02.539765 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.539794 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260618 23:32:02.539884 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.539626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.0.1:37446)
0618 23:32:02.539793 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.539837 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":72,"threads_started":1}
I20260618 23:32:02.540041 7809 leader_election.cc:288] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2270c132876245d0b99153c4cc9f43db (127.0.98.62:43955), 34f7483731cd4088aad3cbc7663639fe (127.0.98.60:40637)
W20260618 23:32:02.540196 7084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
W20260618 23:32:02.540493 7272 leader_election.cc:334] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2270c132876245d0b99153c4cc9f43db (127.0.98.62:43955): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
I20260618 23:32:02.540804 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.540653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.0.1:37468)
0618 23:32:02.540697 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.540749 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260618 23:32:02.540946 7084 leader_election.cc:334] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2270c132876245d0b99153c4cc9f43db (127.0.98.62:43955): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
I20260618 23:32:02.542809 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.538927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48270 (local address 127.0.98.61:36757)
0618 23:32:02.539040 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:02.539043 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:02.539058 (+ 15us) server_negotiation.cc:407] Connection header received
0618 23:32:02.539530 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:02.539532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:02.539574 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:02.539871 (+ 297us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:02.539913 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.540760 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.541912 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.542439 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.542534 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:02.542552 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:02.542716 (+ 164us) server_negotiation.cc:299] Negotiation successful
0618 23:32:02.542758 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":58,"threads_started":1}
I20260618 23:32:02.542850 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.538832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36757 (local address 127.0.0.1:48270)
0618 23:32:02.538985 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.538996 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:02.539065 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:02.539696 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:02.539698 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:02.539702 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:02.539792 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:02.539796 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.540865 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.540865 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:02.541819 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:02.541821 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.542539 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.542540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:02.542624 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:02.542630 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:32:02.542694 (+ 64us) client_negotiation.cc:240] Negotiation successful
0618 23:32:02.542778 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":38,"thread_start_us":91,"threads_started":1}
I20260618 23:32:02.543128 7107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34f7483731cd4088aad3cbc7663639fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "22b5aed081f942d599a7192cba8c03a8" is_pre_election: true
I20260618 23:32:02.543257 7107 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34f7483731cd4088aad3cbc7663639fe in term 1.
I20260618 23:32:02.543473 7271 leader_election.cc:302] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22b5aed081f942d599a7192cba8c03a8, 34f7483731cd4088aad3cbc7663639fe; no voters: 2270c132876245d0b99153c4cc9f43db
I20260618 23:32:02.543694 7808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260618 23:32:02.543788 7808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2270c132876245d0b99153c4cc9f43db)
I20260618 23:32:02.543812 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.539656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40637 (local address 127.0.0.1:35384)
0618 23:32:02.540005 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.540016 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:02.540075 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:02.540341 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:02.540342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:02.540346 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:02.540412 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:02.540414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.541440 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.541441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:02.542530 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:02.542532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.543506 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.543507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:02.543628 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:02.543634 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:32:02.543688 (+ 54us) client_negotiation.cc:240] Negotiation successful
0618 23:32:02.543745 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":64,"threads_started":1}
I20260618 23:32:02.543839 7808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [term 1 FOLLOWER]: Advancing to term 2
I20260618 23:32:02.543790 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.539890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35384 (local address 127.0.98.60:40637)
0618 23:32:02.540195 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:02.540197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:02.540205 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:32:02.540232 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:02.540233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:02.540244 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:02.540515 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:02.540554 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.541342 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.542615 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.543360 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.543451 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:02.543460 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:02.543693 (+ 233us) server_negotiation.cc:299] Negotiation successful
0618 23:32:02.543731 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":112,"threads_started":1}
I20260618 23:32:02.544277 7295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "22b5aed081f942d599a7192cba8c03a8" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "34f7483731cd4088aad3cbc7663639fe" is_pre_election: true
I20260618 23:32:02.544901 7808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: false } }
I20260618 23:32:02.545092 7808 leader_election.cc:288] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [CANDIDATE]: Term 2 election: Requested vote from peers 2270c132876245d0b99153c4cc9f43db (127.0.98.62:43955), 22b5aed081f942d599a7192cba8c03a8 (127.0.98.61:36757)
I20260618 23:32:02.545104 7295 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 22b5aed081f942d599a7192cba8c03a8 in current term 2: Already voted for candidate 34f7483731cd4088aad3cbc7663639fe in this term.
I20260618 23:32:02.545262 7107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "34f7483731cd4088aad3cbc7663639fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "22b5aed081f942d599a7192cba8c03a8"
I20260618 23:32:02.545313 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.545198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.0.1:37484)
0618 23:32:02.545247 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.545287 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260618 23:32:02.545346 7107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [term 1 FOLLOWER]: Advancing to term 2
I20260618 23:32:02.545403 7081 leader_election.cc:302] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 22b5aed081f942d599a7192cba8c03a8; no voters: 2270c132876245d0b99153c4cc9f43db, 34f7483731cd4088aad3cbc7663639fe
I20260618 23:32:02.545714 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.545618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.0.1:37488)
0618 23:32:02.545657 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.545687 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260618 23:32:02.545814 7272 leader_election.cc:334] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2270c132876245d0b99153c4cc9f43db (127.0.98.62:43955): Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
I20260618 23:32:02.546320 7107 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34f7483731cd4088aad3cbc7663639fe in term 2.
I20260618 23:32:02.546391 7809 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 23:32:02.546487 7271 leader_election.cc:302] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22b5aed081f942d599a7192cba8c03a8, 34f7483731cd4088aad3cbc7663639fe; no voters: 2270c132876245d0b99153c4cc9f43db
I20260618 23:32:02.546588 7808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [term 2 FOLLOWER]: Leader election won for term 2
I20260618 23:32:02.546778 7808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [term 2 LEADER]: Becoming Leader. State: Replica: 34f7483731cd4088aad3cbc7663639fe, State: Running, Role: LEADER
I20260618 23:32:02.546854 7808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: false } }
I20260618 23:32:02.547206 7818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34f7483731cd4088aad3cbc7663639fe. Latest consensus state: current_term: 2 leader_uuid: "34f7483731cd4088aad3cbc7663639fe" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: false } } }
I20260618 23:32:02.547287 7818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [sys.catalog]: This master's current role is: LEADER
I20260618 23:32:02.547480 7820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:32:02.548012 7820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:32:02.548106 7820 catalog_manager.cc:1259] Loaded cluster ID: b45c6bc9c18d4dc7bb5d96301b19a73c
I20260618 23:32:02.548139 7820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:32:02.548336 7820 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:32:02.548435 7820 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe: Loaded TSK: 0
I20260618 23:32:02.548589 7820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:32:02.606240 7285 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:32:02.608248 7285 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:32:02.609521 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.609349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.0.1:37520)
0618 23:32:02.609444 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.609495 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260618 23:32:02.609534 7107 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [term 2 FOLLOWER]: Refusing update from remote peer 34f7483731cd4088aad3cbc7663639fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260618 23:32:02.609905 7818 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260618 23:32:02.610055 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.609824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.0.1:37536)
0618 23:32:02.609975 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.610003 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260618 23:32:02.610165 7272 consensus_peers.cc:487] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe -> Peer 2270c132876245d0b99153c4cc9f43db (127.0.98.62:43955): Couldn't send request to peer 2270c132876245d0b99153c4cc9f43db. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260618 23:32:02.610716 7809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34f7483731cd4088aad3cbc7663639fe. Latest consensus state: current_term: 2 leader_uuid: "34f7483731cd4088aad3cbc7663639fe" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: false } } }
I20260618 23:32:02.610836 7809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:32:02.611433 7821 mvcc.cc:205] Tried to move back new op lower bound from 7298357340605825024 to 7298357340352491520. Current Snapshot: MvccSnapshot[applied={T|T < 7298357340605825024}]
I20260618 23:32:02.611737 7808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "34f7483731cd4088aad3cbc7663639fe" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: false } } }
I20260618 23:32:02.611851 7808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe [sys.catalog]: This master's current role is: LEADER
I20260618 23:32:02.611799 7809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "34f7483731cd4088aad3cbc7663639fe" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "2270c132876245d0b99153c4cc9f43db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43955 } } peers { permanent_uuid: "22b5aed081f942d599a7192cba8c03a8" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36757 } attrs { promote: false } } peers { permanent_uuid: "34f7483731cd4088aad3cbc7663639fe" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40637 } attrs { promote: false } } }
I20260618 23:32:02.611892 7809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22b5aed081f942d599a7192cba8c03a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:32:02.611980 7822 mvcc.cc:205] Tried to move back new op lower bound from 7298357340605825024 to 7298357340352491520. Current Snapshot: MvccSnapshot[applied={T|T < 7298357340605825024 or (T in {7298357340605825024})}]
I20260618 23:32:02.617946 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.614172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32993 (local address 127.0.0.1:48132)
0618 23:32:02.614204 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.614217 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:02.614260 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:02.614811 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:02.614812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:02.614815 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:02.614881 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:02.614884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.615781 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.615782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:02.616867 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:02.616870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.617656 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.617657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:02.617727 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:02.617733 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:32:02.617799 (+ 66us) client_negotiation.cc:240] Negotiation successful
0618 23:32:02.617863 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260618 23:32:02.617960 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.614299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48132 (local address 127.0.98.1:32993)
0618 23:32:02.614655 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:02.614657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:02.614666 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:32:02.614703 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:02.614704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:02.614715 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:02.614796 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:02.614968 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.615649 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.616962 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.617561 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.617651 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:02.617670 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:02.617817 (+ 147us) server_negotiation.cc:299] Negotiation successful
0618 23:32:02.617867 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":74,"threads_started":1}
I20260618 23:32:02.618323 7398 tablet_service.cc:1467] Processing CreateTablet for tablet 88a20b41f0ce4d529d5730b8230a6be5 (DEFAULT_TABLE table=test-table [id=50cc29a27cb844cab797442a7a8dc5a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:32:02.618494 7398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88a20b41f0ce4d529d5730b8230a6be5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:02.618753 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.614287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35403 (local address 127.0.0.1:44892)
0618 23:32:02.614386 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.614396 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:02.614440 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:02.615046 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:02.615046 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:02.615049 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:02.615094 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:02.615095 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.616074 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.616075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:02.617767 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:02.617768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.618483 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.618484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:02.618551 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:02.618554 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:32:02.618600 (+ 46us) client_negotiation.cc:240] Negotiation successful
0618 23:32:02.618682 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":403}
I20260618 23:32:02.619052 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.614359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37473 (local address 127.0.0.1:57882)
0618 23:32:02.614492 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.614507 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:02.614572 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:02.614975 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:02.614976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:02.614978 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:02.615039 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:02.615042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.616080 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.616081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:02.617441 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:02.617443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.618792 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.618793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:02.618856 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:02.618862 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:32:02.618910 (+ 48us) client_negotiation.cc:240] Negotiation successful
0618 23:32:02.618973 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":37,"thread_start_us":79,"threads_started":1}
I20260618 23:32:02.619079 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.614382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44892 (local address 127.0.98.2:35403)
0618 23:32:02.614905 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:02.614907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:02.614917 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:32:02.614948 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:02.614949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:02.614963 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:02.615041 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:02.615191 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.615974 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.617852 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.618397 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.618484 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:02.618496 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:02.618958 (+ 462us) server_negotiation.cc:299] Negotiation successful
0618 23:32:02.619006 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":86,"threads_started":1}
I20260618 23:32:02.619304 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.614657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57882 (local address 127.0.98.3:37473)
0618 23:32:02.614784 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:02.614787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:02.614797 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:32:02.614825 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:02.614826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:02.614837 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:02.614910 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:02.615146 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.615979 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.617544 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.618646 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.618730 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:02.618747 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:02.619175 (+ 428us) server_negotiation.cc:299] Negotiation successful
0618 23:32:02.619229 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260618 23:32:02.619406 7535 tablet_service.cc:1467] Processing CreateTablet for tablet 88a20b41f0ce4d529d5730b8230a6be5 (DEFAULT_TABLE table=test-table [id=50cc29a27cb844cab797442a7a8dc5a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:32:02.619617 7535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88a20b41f0ce4d529d5730b8230a6be5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:02.620653 7831 tablet_bootstrap.cc:492] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a: Bootstrap starting.
I20260618 23:32:02.620841 7832 tablet_bootstrap.cc:492] T 88a20b41f0ce4d529d5730b8230a6be5 P 8232a14a2ef3499ea04b5f76d538cf83: Bootstrap starting.
I20260618 23:32:02.621043 7671 tablet_service.cc:1467] Processing CreateTablet for tablet 88a20b41f0ce4d529d5730b8230a6be5 (DEFAULT_TABLE table=test-table [id=50cc29a27cb844cab797442a7a8dc5a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:32:02.621206 7831 tablet_bootstrap.cc:654] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:02.621208 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88a20b41f0ce4d529d5730b8230a6be5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:02.621490 7832 tablet_bootstrap.cc:654] T 88a20b41f0ce4d529d5730b8230a6be5 P 8232a14a2ef3499ea04b5f76d538cf83: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:02.622493 7831 tablet_bootstrap.cc:492] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a: No bootstrap required, opened a new log
I20260618 23:32:02.622576 7831 ts_tablet_manager.cc:1397] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:32:02.622612 7832 tablet_bootstrap.cc:492] T 88a20b41f0ce4d529d5730b8230a6be5 P 8232a14a2ef3499ea04b5f76d538cf83: No bootstrap required, opened a new log
I20260618 23:32:02.622666 7832 ts_tablet_manager.cc:1397] T 88a20b41f0ce4d529d5730b8230a6be5 P 8232a14a2ef3499ea04b5f76d538cf83: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:32:02.622677 7835 tablet_bootstrap.cc:492] T 88a20b41f0ce4d529d5730b8230a6be5 P 0a88d6f8b7bf48ffb3dd2bc35d448cd5: Bootstrap starting.
I20260618 23:32:02.622814 7831 raft_consensus.cc:348] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2015d1fccc3456eae43d83da8ba816a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32993 } } peers { permanent_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37473 } } peers { permanent_uuid: "8232a14a2ef3499ea04b5f76d538cf83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35403 } }
I20260618 23:32:02.622836 7832 raft_consensus.cc:348] T 88a20b41f0ce4d529d5730b8230a6be5 P 8232a14a2ef3499ea04b5f76d538cf83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2015d1fccc3456eae43d83da8ba816a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32993 } } peers { permanent_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37473 } } peers { permanent_uuid: "8232a14a2ef3499ea04b5f76d538cf83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35403 } }
I20260618 23:32:02.622915 7831 raft_consensus.cc:374] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:02.622915 7832 raft_consensus.cc:374] T 88a20b41f0ce4d529d5730b8230a6be5 P 8232a14a2ef3499ea04b5f76d538cf83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:02.622954 7832 raft_consensus.cc:729] T 88a20b41f0ce4d529d5730b8230a6be5 P 8232a14a2ef3499ea04b5f76d538cf83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8232a14a2ef3499ea04b5f76d538cf83, State: Initialized, Role: FOLLOWER
I20260618 23:32:02.622954 7831 raft_consensus.cc:729] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2015d1fccc3456eae43d83da8ba816a, State: Initialized, Role: FOLLOWER
I20260618 23:32:02.623018 7832 consensus_queue.cc:260] T 88a20b41f0ce4d529d5730b8230a6be5 P 8232a14a2ef3499ea04b5f76d538cf83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2015d1fccc3456eae43d83da8ba816a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32993 } } peers { permanent_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37473 } } peers { permanent_uuid: "8232a14a2ef3499ea04b5f76d538cf83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35403 } }
I20260618 23:32:02.623041 7831 consensus_queue.cc:260] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2015d1fccc3456eae43d83da8ba816a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32993 } } peers { permanent_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37473 } } peers { permanent_uuid: "8232a14a2ef3499ea04b5f76d538cf83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35403 } }
I20260618 23:32:02.623229 7835 tablet_bootstrap.cc:654] T 88a20b41f0ce4d529d5730b8230a6be5 P 0a88d6f8b7bf48ffb3dd2bc35d448cd5: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:02.623489 7832 ts_tablet_manager.cc:1428] T 88a20b41f0ce4d529d5730b8230a6be5 P 8232a14a2ef3499ea04b5f76d538cf83: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:02.623499 7831 ts_tablet_manager.cc:1428] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:02.624064 7835 tablet_bootstrap.cc:492] T 88a20b41f0ce4d529d5730b8230a6be5 P 0a88d6f8b7bf48ffb3dd2bc35d448cd5: No bootstrap required, opened a new log
I20260618 23:32:02.624116 7835 ts_tablet_manager.cc:1397] T 88a20b41f0ce4d529d5730b8230a6be5 P 0a88d6f8b7bf48ffb3dd2bc35d448cd5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:02.624240 7835 raft_consensus.cc:348] T 88a20b41f0ce4d529d5730b8230a6be5 P 0a88d6f8b7bf48ffb3dd2bc35d448cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2015d1fccc3456eae43d83da8ba816a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32993 } } peers { permanent_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37473 } } peers { permanent_uuid: "8232a14a2ef3499ea04b5f76d538cf83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35403 } }
I20260618 23:32:02.624316 7835 raft_consensus.cc:374] T 88a20b41f0ce4d529d5730b8230a6be5 P 0a88d6f8b7bf48ffb3dd2bc35d448cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:02.624341 7835 raft_consensus.cc:729] T 88a20b41f0ce4d529d5730b8230a6be5 P 0a88d6f8b7bf48ffb3dd2bc35d448cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a88d6f8b7bf48ffb3dd2bc35d448cd5, State: Initialized, Role: FOLLOWER
I20260618 23:32:02.624396 7835 consensus_queue.cc:260] T 88a20b41f0ce4d529d5730b8230a6be5 P 0a88d6f8b7bf48ffb3dd2bc35d448cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2015d1fccc3456eae43d83da8ba816a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32993 } } peers { permanent_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37473 } } peers { permanent_uuid: "8232a14a2ef3499ea04b5f76d538cf83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35403 } }
I20260618 23:32:02.624563 7835 ts_tablet_manager.cc:1428] T 88a20b41f0ce4d529d5730b8230a6be5 P 0a88d6f8b7bf48ffb3dd2bc35d448cd5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:02.624614 7601 heartbeater.cc:503] Master 127.0.98.60:40637 was elected leader, sending a full tablet report...
I20260618 23:32:02.624881 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.624019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.1:33651)
0618 23:32:02.624771 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.624825 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":720,"thread_start_us":102,"threads_started":1}
I20260618 23:32:02.625032 7464 heartbeater.cc:503] Master 127.0.98.60:40637 was elected leader, sending a full tablet report...
I20260618 23:32:02.625500 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.624533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.2:53613)
0618 23:32:02.625417 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.625457 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":859,"thread_start_us":86,"threads_started":1}
I20260618 23:32:02.625618 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.625210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.1:36689)
0618 23:32:02.625572 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.625597 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343}
I20260618 23:32:02.625852 7737 heartbeater.cc:503] Master 127.0.98.60:40637 was elected leader, sending a full tablet report...
I20260618 23:32:02.625918 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.625733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.2:42905)
0618 23:32:02.625866 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.625892 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260618 23:32:02.626035 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.624924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.3:60307)
0618 23:32:02.625790 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.625834 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":829,"thread_start_us":78,"threads_started":1}
I20260618 23:32:02.626708 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.626620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.3:46613)
0618 23:32:02.626652 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.626677 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260618 23:32:02.783175 7837 raft_consensus.cc:482] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:32:02.783350 7837 raft_consensus.cc:504] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2015d1fccc3456eae43d83da8ba816a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32993 } } peers { permanent_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37473 } } peers { permanent_uuid: "8232a14a2ef3499ea04b5f76d538cf83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35403 } }
I20260618 23:32:02.783810 7837 leader_election.cc:288] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a88d6f8b7bf48ffb3dd2bc35d448cd5 (127.0.98.3:37473), 8232a14a2ef3499ea04b5f76d538cf83 (127.0.98.2:35403)
I20260618 23:32:02.787478 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.784284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43973 (local address 127.0.98.2:35403)
0618 23:32:02.784352 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:02.784354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:02.784365 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:32:02.784437 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:02.784438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:02.784450 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:02.784524 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:02.784702 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.785283 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.786522 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.787062 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.787125 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:02.787136 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:02.787342 (+ 206us) server_negotiation.cc:299] Negotiation successful
0618 23:32:02.787375 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 23:32:02.787499 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.784128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35403 (local address 127.0.98.1:43973)
0618 23:32:02.784240 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.784257 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:02.784361 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:02.784529 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:02.784530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:02.784534 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:02.784618 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:02.784620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.785400 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.785401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:02.786401 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:02.786403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.787158 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.787159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:02.787243 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:02.787249 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:32:02.787317 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 23:32:02.787368 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260618 23:32:02.787886 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.784268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45355 (local address 127.0.98.3:37473)
0618 23:32:02.784338 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:02.784340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:02.784533 (+ 193us) server_negotiation.cc:407] Connection header received
0618 23:32:02.784648 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:02.784649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:02.784662 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:02.784740 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:02.784917 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.785672 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.786890 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.787380 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.787451 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:02.787464 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:02.787743 (+ 279us) server_negotiation.cc:299] Negotiation successful
0618 23:32:02.787813 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260618 23:32:02.788148 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.784128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37473 (local address 127.0.98.1:45355)
0618 23:32:02.784443 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.784454 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:02.784552 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:02.784745 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:02.784746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:02.784748 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:02.784839 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:02.784841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.785803 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.785805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:02.786780 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:02.786782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.787511 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.787512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:02.787617 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:02.787623 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:32:02.788000 (+ 377us) client_negotiation.cc:240] Negotiation successful
0618 23:32:02.788054 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":113,"thread_start_us":134,"threads_started":1}
I20260618 23:32:02.788069 7555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a20b41f0ce4d529d5730b8230a6be5" candidate_uuid: "e2015d1fccc3456eae43d83da8ba816a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8232a14a2ef3499ea04b5f76d538cf83" is_pre_election: true
I20260618 23:32:02.788226 7555 raft_consensus.cc:2454] T 88a20b41f0ce4d529d5730b8230a6be5 P 8232a14a2ef3499ea04b5f76d538cf83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2015d1fccc3456eae43d83da8ba816a in term 0.
I20260618 23:32:02.788508 7352 leader_election.cc:302] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8232a14a2ef3499ea04b5f76d538cf83, e2015d1fccc3456eae43d83da8ba816a; no voters:
I20260618 23:32:02.788533 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a20b41f0ce4d529d5730b8230a6be5" candidate_uuid: "e2015d1fccc3456eae43d83da8ba816a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" is_pre_election: true
I20260618 23:32:02.788615 7691 raft_consensus.cc:2454] T 88a20b41f0ce4d529d5730b8230a6be5 P 0a88d6f8b7bf48ffb3dd2bc35d448cd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2015d1fccc3456eae43d83da8ba816a in term 0.
I20260618 23:32:02.788686 7837 raft_consensus.cc:2781] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:32:02.788735 7837 raft_consensus.cc:482] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:32:02.788765 7837 raft_consensus.cc:3037] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:02.789669 7837 raft_consensus.cc:504] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2015d1fccc3456eae43d83da8ba816a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32993 } } peers { permanent_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37473 } } peers { permanent_uuid: "8232a14a2ef3499ea04b5f76d538cf83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35403 } }
I20260618 23:32:02.789934 7837 leader_election.cc:288] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a [CANDIDATE]: Term 1 election: Requested vote from peers 0a88d6f8b7bf48ffb3dd2bc35d448cd5 (127.0.98.3:37473), 8232a14a2ef3499ea04b5f76d538cf83 (127.0.98.2:35403)
I20260618 23:32:02.790020 7691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a20b41f0ce4d529d5730b8230a6be5" candidate_uuid: "e2015d1fccc3456eae43d83da8ba816a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5"
I20260618 23:32:02.790091 7691 raft_consensus.cc:3037] T 88a20b41f0ce4d529d5730b8230a6be5 P 0a88d6f8b7bf48ffb3dd2bc35d448cd5 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:02.790041 7555 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a20b41f0ce4d529d5730b8230a6be5" candidate_uuid: "e2015d1fccc3456eae43d83da8ba816a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8232a14a2ef3499ea04b5f76d538cf83"
I20260618 23:32:02.790129 7555 raft_consensus.cc:3037] T 88a20b41f0ce4d529d5730b8230a6be5 P 8232a14a2ef3499ea04b5f76d538cf83 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:02.790920 7555 raft_consensus.cc:2454] T 88a20b41f0ce4d529d5730b8230a6be5 P 8232a14a2ef3499ea04b5f76d538cf83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2015d1fccc3456eae43d83da8ba816a in term 1.
I20260618 23:32:02.790936 7691 raft_consensus.cc:2454] T 88a20b41f0ce4d529d5730b8230a6be5 P 0a88d6f8b7bf48ffb3dd2bc35d448cd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2015d1fccc3456eae43d83da8ba816a in term 1.
I20260618 23:32:02.791177 7353 leader_election.cc:302] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a88d6f8b7bf48ffb3dd2bc35d448cd5, e2015d1fccc3456eae43d83da8ba816a; no voters:
I20260618 23:32:02.791301 7837 raft_consensus.cc:2781] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:32:02.791491 7837 raft_consensus.cc:686] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a [term 1 LEADER]: Becoming Leader. State: Replica: e2015d1fccc3456eae43d83da8ba816a, State: Running, Role: LEADER
I20260618 23:32:02.791601 7837 consensus_queue.cc:237] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2015d1fccc3456eae43d83da8ba816a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32993 } } peers { permanent_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37473 } } peers { permanent_uuid: "8232a14a2ef3499ea04b5f76d538cf83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35403 } }
I20260618 23:32:02.791859 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.791751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.1:56209)
0618 23:32:02.791805 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.791830 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260618 23:32:02.792416 7285 catalog_manager.cc:5495] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a reported cstate change: term changed from 0 to 1, leader changed from <none> to e2015d1fccc3456eae43d83da8ba816a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e2015d1fccc3456eae43d83da8ba816a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2015d1fccc3456eae43d83da8ba816a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8232a14a2ef3499ea04b5f76d538cf83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35403 } health_report { overall_health: UNKNOWN } } }
I20260618 23:32:02.805761 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.802859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48146 (local address 127.0.98.1:32993)
0618 23:32:02.802939 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:02.802941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:02.802950 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:32:02.802996 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:02.802997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:02.803021 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:02.803094 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:32:02.803266 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.803874 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.804670 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:02.804904 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:02.804982 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:02.805307 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:32:02.805434 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:32:02.805512 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:02.805648 (+ 136us) server_negotiation.cc:299] Negotiation successful
0618 23:32:02.805680 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 23:32:02.807498 7691 raft_consensus.cc:1261] T 88a20b41f0ce4d529d5730b8230a6be5 P 0a88d6f8b7bf48ffb3dd2bc35d448cd5 [term 1 FOLLOWER]: Refusing update from remote peer e2015d1fccc3456eae43d83da8ba816a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:32:02.807885 7555 raft_consensus.cc:1261] T 88a20b41f0ce4d529d5730b8230a6be5 P 8232a14a2ef3499ea04b5f76d538cf83 [term 1 FOLLOWER]: Refusing update from remote peer e2015d1fccc3456eae43d83da8ba816a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:32:02.808313 7837 consensus_queue.cc:1035] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8232a14a2ef3499ea04b5f76d538cf83" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:32:02.808497 7845 consensus_queue.cc:1035] T 88a20b41f0ce4d529d5730b8230a6be5 P e2015d1fccc3456eae43d83da8ba816a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a88d6f8b7bf48ffb3dd2bc35d448cd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:32:02.809321 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.809191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.3:55127)
0618 23:32:02.809244 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.809279 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260618 23:32:02.809815 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.809648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.2:40381)
0618 23:32:02.809726 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.809774 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260618 23:32:02.811069 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.810963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.1:39417)
0618 23:32:02.811004 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.811030 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260618 23:32:02.811403 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.811228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.3:52021)
0618 23:32:02.811347 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.811371 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260618 23:32:02.811729 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.811652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.1:59965)
0618 23:32:02.811686 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.811707 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260618 23:32:02.812703 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:02.812569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43955 (local address 127.0.98.2:58081)
0618 23:32:02.812643 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.812671 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:43955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260618 23:32:02.815268 7285 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:32:02.815384 7285 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:32:02.817121 7285 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34f7483731cd4088aad3cbc7663639fe: Sending DeleteTablet for 3 replicas of tablet 88a20b41f0ce4d529d5730b8230a6be5
I20260618 23:32:02.817519 7671 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a20b41f0ce4d529d5730b8230a6be5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:32:02 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:57882
I20260618 23:32:02.817535 7398 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a20b41f0ce4d529d5730b8230a6be5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:32:02 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48132
I20260618 23:32:02.817770 7859 tablet_replica.cc:330] T 88a20b41f0ce4d529d5730b8230a6be5 P 0a88d6f8b7bf48ffb3dd2bc35d448cd5: stopping tablet replica
I20260618 23:32:02.817838 7859 raft_consensus.cc:2229] T 88a20b41f0ce4d529d5730b8230a6be5 P 0a88d6f8b7bf48ffb3dd2bc35d448cd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:32:02.817905 7859 raft_consensus.cc:2258] T 88a20b41f0ce4d529d5730b8230a6be5 P 0a88d6f8b7bf48ffb3dd2bc35d448cd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:32:02.818010 7535 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a20b41f0ce4d529d5730b8230a6be5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:32:02 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:44892
I20260618 23:32:02.818325 7859 ts_tablet_manager.cc:1905] T 88a20b41f0ce4d529d5730b8230a6be5 P 0a88d6f8b7bf48ffb3dd2bc35d448cd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:32:02.818413 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 7244
I20260618 23:32:02.820441 7859 ts_tablet_manager.cc:1918] T 88a20b41f0ce4d529d5730b8230a6be5 P 0a88d6f8b7bf48ffb3dd2bc35d448cd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:32:02.820499 7859 log.cc:1196] T 88a20b41f0ce4d529d5730b8230a6be5 P 0a88d6f8b7bf48ffb3dd2bc35d448cd5: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/88a20b41f0ce4d529d5730b8230a6be5
I20260618 23:32:02.820791 7859 ts_tablet_manager.cc:1939] T 88a20b41f0ce4d529d5730b8230a6be5 P 0a88d6f8b7bf48ffb3dd2bc35d448cd5: Deleting consensus metadata
I20260618 23:32:02.820931 7861 tablet_replica.cc:330] T 88a20b41f0ce4d529d5730b8230a6be5 P 8232a14a2ef3499ea04b5f76d538cf83: stopping tablet replica
I20260618 23:32:02.821034 7861 raft_consensus.cc:2229] T 88a20b41f0ce4d529d5730b8230a6be5 P 8232a14a2ef3499ea04b5f76d538cf83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:32:02.821099 7861 raft_consensus.cc:2258] T 88a20b41f0ce4d529d5730b8230a6be5 P 8232a14a2ef3499ea04b5f76d538cf83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:32:02.821452 7271 catalog_manager.cc:4841] TS 0a88d6f8b7bf48ffb3dd2bc35d448cd5 (127.0.98.3:37473): tablet 88a20b41f0ce4d529d5730b8230a6be5 (table test-table [id=50cc29a27cb844cab797442a7a8dc5a6]) successfully deleted
I20260618 23:32:02.821514 7861 ts_tablet_manager.cc:1905] T 88a20b41f0ce4d529d5730b8230a6be5 P 8232a14a2ef3499ea04b5f76d538cf83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:32:02.823642 7861 ts_tablet_manager.cc:1918] T 88a20b41f0ce4d529d5730b8230a6be5 P 8232a14a2ef3499ea04b5f76d538cf83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:32:02.823714 7861 log.cc:1196] T 88a20b41f0ce4d529d5730b8230a6be5 P 8232a14a2ef3499ea04b5f76d538cf83: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/88a20b41f0ce4d529d5730b8230a6be5
I20260618 23:32:02.823976 7861 ts_tablet_manager.cc:1939] T 88a20b41f0ce4d529d5730b8230a6be5 P 8232a14a2ef3499ea04b5f76d538cf83: Deleting consensus metadata
I20260618 23:32:02.824350 7270 catalog_manager.cc:4841] TS 8232a14a2ef3499ea04b5f76d538cf83 (127.0.98.2:35403): tablet 88a20b41f0ce4d529d5730b8230a6be5 (table test-table [id=50cc29a27cb844cab797442a7a8dc5a6]) successfully deleted
W20260618 23:32:02.825764 7828 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:02.825602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32993 (local address 127.0.0.1:48158)
0618 23:32:02.825659 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:02.825694 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32993: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":30}
I20260618 23:32:02.825909 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 7472
W20260618 23:32:02.828379 7270 catalog_manager.cc:4584] TS e2015d1fccc3456eae43d83da8ba816a (127.0.98.1:32993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 88a20b41f0ce4d529d5730b8230a6be5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32993: connect: Connection reset by peer (error 104)
I20260618 23:32:02.832837 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 7608
I20260618 23:32:02.838690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 6963
I20260618 23:32:02.844702 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 7151
2026-06-18T23:32:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5197 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:32:02 dist-test-slave-6224 krb5kdc[7864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:32:02 dist-test-slave-6224 krb5kdc[7864](info): set up 2 sockets
Jun 18 23:32:02 dist-test-slave-6224 krb5kdc[7864](info): commencing operation
krb5kdc: starting...
W20260618 23:32:03.892295 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:32:03 dist-test-slave-6224 krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:32:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:32:03Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260618 23:32:04.020107 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40723
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:32:04.105379 7877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:04.105597 7877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:04.105619 7877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:04.106778 7877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:32:04.106825 7877 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260618 23:32:04.106839 7877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:04.106853 7877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:04.106865 7877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:32:04.106925 7877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:32:04.108453 7877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40723
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:04.108707 7877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:04.108958 7877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:04.114205 7883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:04.114179 7885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:04.114248 7882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:04.114367 7877 server_base.cc:1034] running on GCE node
I20260618 23:32:04.114705 7877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:04.114960 7877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:04.116139 7877 hybrid_clock.cc:648] HybridClock initialized: now 1781825524116127 us; error 34 us; skew 500 ppm
Jun 18 23:32:04 dist-test-slave-6224 krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825524, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:32:04.122845 7877 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260618 23:32:04.123384 7877 webserver.cc:460] Webserver started at http://127.0.98.62:39587/ using document root <none> and password file <none>
I20260618 23:32:04.123690 7877 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:04.123760 7877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:04.123910 7877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:32:04.125042 7877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "efdfd46124e04c92bb3c500fe404e02f"
format_stamp: "Formatted at 2026-06-18 23:32:04 on dist-test-slave-6224"
server_key: "7b149b7579b1c8198acfc9fcc3f82fa3"
server_key_iv: "35e0d5643718c43350b3aa7fff0f3779"
server_key_version: "encryptionkey@0"
I20260618 23:32:04.125403 7877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "efdfd46124e04c92bb3c500fe404e02f"
format_stamp: "Formatted at 2026-06-18 23:32:04 on dist-test-slave-6224"
server_key: "7b149b7579b1c8198acfc9fcc3f82fa3"
server_key_iv: "35e0d5643718c43350b3aa7fff0f3779"
server_key_version: "encryptionkey@0"
I20260618 23:32:04.126873 7877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:32:04.127805 7892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:04.128070 7877 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:32:04.128188 7877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "efdfd46124e04c92bb3c500fe404e02f"
format_stamp: "Formatted at 2026-06-18 23:32:04 on dist-test-slave-6224"
server_key: "7b149b7579b1c8198acfc9fcc3f82fa3"
server_key_iv: "35e0d5643718c43350b3aa7fff0f3779"
server_key_version: "encryptionkey@0"
I20260618 23:32:04.128264 7877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:04.159476 7877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:04.161767 7877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:04.161932 7877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:04.167233 7877 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40723
I20260618 23:32:04.167296 7954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40723 every 8 connection(s)
I20260618 23:32:04.167694 7877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:32:04.168546 7955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:04.171387 7955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f: Bootstrap starting.
I20260618 23:32:04.172135 7955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:04.172497 7955 log.cc:824] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f: Log is configured to *not* fsync() on all Append() calls
I20260618 23:32:04.173349 7955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f: No bootstrap required, opened a new log
I20260618 23:32:04.175216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 7877
I20260618 23:32:04.175300 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:32:04.175588 392 external_mini_cluster.cc:1428] Setting key 513eb15f539be233a0e5e3d6e9d20589
I20260618 23:32:04.176240 7955 raft_consensus.cc:348] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efdfd46124e04c92bb3c500fe404e02f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40723 } }
I20260618 23:32:04.176486 7955 raft_consensus.cc:374] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:04.176529 7955 raft_consensus.cc:729] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efdfd46124e04c92bb3c500fe404e02f, State: Initialized, Role: FOLLOWER
I20260618 23:32:04.176688 7955 consensus_queue.cc:260] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efdfd46124e04c92bb3c500fe404e02f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40723 } }
I20260618 23:32:04.176782 7955 raft_consensus.cc:388] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:32:04.176815 7955 raft_consensus.cc:482] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:32:04.176883 7955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:04.177812 7955 raft_consensus.cc:504] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efdfd46124e04c92bb3c500fe404e02f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40723 } }
I20260618 23:32:04.177942 7955 leader_election.cc:302] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efdfd46124e04c92bb3c500fe404e02f; no voters:
I20260618 23:32:04.178150 7955 leader_election.cc:288] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:32:04.178237 7960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:32:04.178460 7955 sys_catalog.cc:564] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:32:04.178484 7960 raft_consensus.cc:686] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f [term 1 LEADER]: Becoming Leader. State: Replica: efdfd46124e04c92bb3c500fe404e02f, State: Running, Role: LEADER
I20260618 23:32:04.178799 7960 consensus_queue.cc:237] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efdfd46124e04c92bb3c500fe404e02f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40723 } }
I20260618 23:32:04.179275 7961 sys_catalog.cc:455] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efdfd46124e04c92bb3c500fe404e02f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efdfd46124e04c92bb3c500fe404e02f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40723 } } }
I20260618 23:32:04.179365 7961 sys_catalog.cc:458] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f [sys.catalog]: This master's current role is: LEADER
I20260618 23:32:04.179842 7962 sys_catalog.cc:455] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f [sys.catalog]: SysCatalogTable state changed. Reason: New leader efdfd46124e04c92bb3c500fe404e02f. Latest consensus state: current_term: 1 leader_uuid: "efdfd46124e04c92bb3c500fe404e02f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efdfd46124e04c92bb3c500fe404e02f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40723 } } }
I20260618 23:32:04.179910 7962 sys_catalog.cc:458] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f [sys.catalog]: This master's current role is: LEADER
I20260618 23:32:04.180389 7969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:32:04.181171 7969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:32:04.182992 7969 catalog_manager.cc:1347] Generated new cluster ID: 248b27063b7d436faf8dedee9c032e30
I20260618 23:32:04.183046 7969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 18 23:32:04 dist-test-slave-6224 krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825523, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260618 23:32:04.188400 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:04.176277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59586 (local address 127.0.98.62:40723)
0618 23:32:04.176533 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:04.176545 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:04.176586 (+ 41us) server_negotiation.cc:407] Connection header received
0618 23:32:04.176735 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:04.176743 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:04.176811 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:04.177013 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:04.177648 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.179186 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.180353 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.181041 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.181163 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:04.184415 (+ 3252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:04.184419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:04.184420 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:04.184433 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:04.186559 (+ 2126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:04.187056 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:04.187059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:04.187060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:04.187184 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:04.187431 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:04.187432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:04.187432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:04.187764 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:04.187921 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:04.188066 (+ 145us) server_negotiation.cc:299] Negotiation successful
0618 23:32:04.188203 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":117,"threads_started":1}
I20260618 23:32:04.210911 7969 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:32:04.211820 7969 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:32:04.218429 7969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f: Generated new TSK 0
I20260618 23:32:04.218669 7969 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260618 23:32:04.250380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40723
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:32:04.332866 7981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:04.333094 7981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:04.333119 7981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:04.334268 7981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:04.334312 7981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:04.334333 7981 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 23:32:04.334352 7981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:32:04.335855 7981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:04.336054 7981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:04.336232 7981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:04.341737 7989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:04.341931 7981 server_base.cc:1034] running on GCE node
W20260618 23:32:04.341787 7986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:04.341750 7987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:04.342272 7981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:04.342628 7981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:04.343803 7981 hybrid_clock.cc:648] HybridClock initialized: now 1781825524343796 us; error 62 us; skew 500 ppm
Jun 18 23:32:04 dist-test-slave-6224 krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825524, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:32:04.346688 7981 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260618 23:32:04.347239 7981 webserver.cc:460] Webserver started at http://127.0.98.1:43905/ using document root <none> and password file <none>
I20260618 23:32:04.347509 7981 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:04.347607 7981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:04.347777 7981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:32:04.348968 7981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "20b9016f98a5411bb034d5e44b4dd52f"
format_stamp: "Formatted at 2026-06-18 23:32:04 on dist-test-slave-6224"
server_key: "2f95ed9721900da52b0dd1582a675d5e"
server_key_iv: "ec414a16a56ab5da126d0903c7fcd485"
server_key_version: "encryptionkey@0"
I20260618 23:32:04.349366 7981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "20b9016f98a5411bb034d5e44b4dd52f"
format_stamp: "Formatted at 2026-06-18 23:32:04 on dist-test-slave-6224"
server_key: "2f95ed9721900da52b0dd1582a675d5e"
server_key_iv: "ec414a16a56ab5da126d0903c7fcd485"
server_key_version: "encryptionkey@0"
I20260618 23:32:04.350836 7981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:32:04.351742 7996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:04.351964 7981 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:32:04.352079 7981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "20b9016f98a5411bb034d5e44b4dd52f"
format_stamp: "Formatted at 2026-06-18 23:32:04 on dist-test-slave-6224"
server_key: "2f95ed9721900da52b0dd1582a675d5e"
server_key_iv: "ec414a16a56ab5da126d0903c7fcd485"
server_key_version: "encryptionkey@0"
I20260618 23:32:04.352161 7981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:04.366992 7981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:04.370347 7981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:04.370519 7981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:04.371114 7981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:32:04.371158 7981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:04.371186 7981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:32:04.371201 7981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 23:32:04 dist-test-slave-6224 krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825524, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260618 23:32:04.379913 7981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36853
I20260618 23:32:04.380043 8111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36853 every 8 connection(s)
I20260618 23:32:04.380311 7981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:32:04.383404 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:04.371908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49899 (local address 127.0.98.62:40723)
0618 23:32:04.372002 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:04.372005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:04.372283 (+ 278us) server_negotiation.cc:407] Connection header received
0618 23:32:04.373124 (+ 841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:04.373126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:04.373153 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:04.373221 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:04.374255 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.375105 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.376904 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.377264 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.377360 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:04.380506 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:04.380508 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:04.380508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:04.380522 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:04.382016 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:04.382447 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:04.382448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:04.382449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:04.382516 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:04.382792 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:04.382793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:04.382794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:04.382934 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:04.383004 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:04.383238 (+ 234us) server_negotiation.cc:299] Negotiation successful
0618 23:32:04.383328 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 23:32:04.383594 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:04.371870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40723 (local address 127.0.98.1:49899)
0618 23:32:04.372111 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:04.372164 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:04.372982 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:04.373352 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:04.373358 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:04.373774 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:04.374119 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:04.374126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.375245 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.375246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:04.376781 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:04.376784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.377380 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.377381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:04.377544 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:04.378187 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:04.378199 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:04.380376 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:04.382102 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:04.382108 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:04.382110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:04.382344 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:04.382606 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:04.382606 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:04.382607 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:04.382700 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:04.383007 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:04.383011 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:04.383123 (+ 112us) client_negotiation.cc:769] Sending connection context
0618 23:32:04.383209 (+ 86us) client_negotiation.cc:240] Negotiation successful
0618 23:32:04.383337 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":157,"threads_started":1}
I20260618 23:32:04.384246 8112 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40723
I20260618 23:32:04.384330 8112 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:04.384511 8112 heartbeater.cc:511] Master 127.0.98.62:40723 requested a full tablet report, sending...
I20260618 23:32:04.385159 7909 ts_manager.cc:194] Registered new tserver with Master: 20b9016f98a5411bb034d5e44b4dd52f (127.0.98.1:36853)
I20260618 23:32:04.385489 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 7981
I20260618 23:32:04.385615 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:32:04.385818 392 external_mini_cluster.cc:1428] Setting key 05bfc7bd0bba278f0127fb72004d7774
I20260618 23:32:04.386165 7909 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49899
I20260618 23:32:04.392202 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:04.385507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34089 (local address 127.0.98.62:40723)
0618 23:32:04.385612 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:04.385615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:04.385716 (+ 101us) server_negotiation.cc:407] Connection header received
0618 23:32:04.385792 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:04.385793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:04.385804 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:04.385862 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:04.386338 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.386974 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.387804 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.388105 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.388188 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:04.389166 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:04.389167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:04.389168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:04.389173 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:04.390779 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:04.391237 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:04.391238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:04.391238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:04.391307 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:04.391641 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:04.391643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:04.391643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:04.391777 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:04.391851 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:04.392049 (+ 198us) server_negotiation.cc:299] Negotiation successful
0618 23:32:04.392129 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260618 23:32:04.392287 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:04.385460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40723 (local address 127.0.98.1:34089)
0618 23:32:04.385651 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:04.385664 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:04.385734 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:04.385865 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:04.385866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:04.386130 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:04.386223 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:04.386226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.387077 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.387078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:04.387722 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:04.387724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.388189 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.388191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:04.388257 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:04.388719 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:04.388724 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:04.389027 (+ 303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:04.390878 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:04.390879 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:04.390879 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:04.391120 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:04.391399 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:04.391400 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:04.391401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:04.391485 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:04.391866 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:04.391867 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:04.391925 (+ 58us) client_negotiation.cc:769] Sending connection context
0618 23:32:04.392029 (+ 104us) client_negotiation.cc:240] Negotiation successful
0618 23:32:04.392160 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260618 23:32:04.410359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40723
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:32:04.493602 8121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:04.493810 8121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:04.493834 8121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:04.494899 8121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:04.494941 8121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:04.494961 8121 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 23:32:04.494979 8121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:32:04.496467 8121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:04.496656 8121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:04.496825 8121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:04.502251 8126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:04.502336 8129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:04.502251 8127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:04.502588 8121 server_base.cc:1034] running on GCE node
I20260618 23:32:04.502869 8121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:04.503115 8121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:04.504266 8121 hybrid_clock.cc:648] HybridClock initialized: now 1781825524504215 us; error 102 us; skew 500 ppm
Jun 18 23:32:04 dist-test-slave-6224 krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825524, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:32:04.507133 8121 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260618 23:32:04.507671 8121 webserver.cc:460] Webserver started at http://127.0.98.2:33223/ using document root <none> and password file <none>
I20260618 23:32:04.507920 8121 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:04.507977 8121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:04.508105 8121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:32:04.509151 8121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "e92c821b3841414c965660b54194112f"
format_stamp: "Formatted at 2026-06-18 23:32:04 on dist-test-slave-6224"
server_key: "a4618ead25dcd314c0b4bfd4423cf704"
server_key_iv: "0a37f070fd3ab5e2823c94a0443b32f6"
server_key_version: "encryptionkey@0"
I20260618 23:32:04.509527 8121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e92c821b3841414c965660b54194112f"
format_stamp: "Formatted at 2026-06-18 23:32:04 on dist-test-slave-6224"
server_key: "a4618ead25dcd314c0b4bfd4423cf704"
server_key_iv: "0a37f070fd3ab5e2823c94a0443b32f6"
server_key_version: "encryptionkey@0"
I20260618 23:32:04.510876 8121 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:32:04.511734 8136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:04.512027 8121 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:04.512125 8121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "e92c821b3841414c965660b54194112f"
format_stamp: "Formatted at 2026-06-18 23:32:04 on dist-test-slave-6224"
server_key: "a4618ead25dcd314c0b4bfd4423cf704"
server_key_iv: "0a37f070fd3ab5e2823c94a0443b32f6"
server_key_version: "encryptionkey@0"
I20260618 23:32:04.512207 8121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:04.534039 8121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:04.536247 8121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:04.536413 8121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:04.537088 8121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:32:04.537146 8121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:04.537177 8121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:32:04.537199 8121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 23:32:04 dist-test-slave-6224 krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825524, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260618 23:32:04.545477 8121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40739
I20260618 23:32:04.545606 8251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40739 every 8 connection(s)
I20260618 23:32:04.545990 8121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:32:04.548712 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:04.537710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52631 (local address 127.0.98.62:40723)
0618 23:32:04.537784 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:04.537787 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:04.538163 (+ 376us) server_negotiation.cc:407] Connection header received
0618 23:32:04.538994 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:04.538997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:04.539011 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:04.539051 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:04.540127 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.540742 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.542413 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.542664 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.542768 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:04.545733 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:04.545735 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:04.545736 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:04.545745 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:04.547247 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:04.547790 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:04.547791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:04.547791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:04.547853 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:04.548101 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:04.548101 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:04.548102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:04.548233 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:04.548295 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:04.548560 (+ 265us) server_negotiation.cc:299] Negotiation successful
0618 23:32:04.548638 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 23:32:04.548797 8145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:04.537735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40723 (local address 127.0.98.2:52631)
0618 23:32:04.537977 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:04.538037 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:04.538823 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:04.539274 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:04.539279 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:04.539705 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:04.540005 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:04.540011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.540876 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.540877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:04.542306 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:04.542310 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.542797 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.542798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:04.542977 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:04.543709 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:04.543720 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:04.545608 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:04.547335 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:04.547340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:04.547341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:04.547703 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:04.547930 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:04.547931 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:04.547931 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:04.548030 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:04.548298 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:04.548301 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:04.548415 (+ 114us) client_negotiation.cc:769] Sending connection context
0618 23:32:04.548514 (+ 99us) client_negotiation.cc:240] Negotiation successful
0618 23:32:04.548645 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":124,"threads_started":1}
I20260618 23:32:04.549458 8252 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40723
I20260618 23:32:04.549551 8252 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:04.549731 8252 heartbeater.cc:511] Master 127.0.98.62:40723 requested a full tablet report, sending...
I20260618 23:32:04.550254 7909 ts_manager.cc:194] Registered new tserver with Master: e92c821b3841414c965660b54194112f (127.0.98.2:40739)
I20260618 23:32:04.550815 7909 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52631
I20260618 23:32:04.556188 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 8121
I20260618 23:32:04.556357 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:32:04.556591 8258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:04.550641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40723 (local address 127.0.98.2:56937)
0618 23:32:04.551029 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:04.551044 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:04.551117 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:04.551272 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:04.551273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:04.551484 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:04.551597 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:04.551599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.552265 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.552266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:04.552812 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:04.552814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.553194 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.553195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:04.553257 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:04.553688 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:04.553692 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:04.553954 (+ 262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:04.555419 (+ 1465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:04.555420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:04.555420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:04.555677 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:04.555887 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:04.555888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:04.555888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:04.555975 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:04.556259 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:04.556260 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:04.556303 (+ 43us) client_negotiation.cc:769] Sending connection context
0618 23:32:04.556378 (+ 75us) client_negotiation.cc:240] Negotiation successful
0618 23:32:04.556494 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":95,"threads_started":1}
I20260618 23:32:04.556619 392 external_mini_cluster.cc:1428] Setting key 8e4ba4870ff6f93eea9e95fe6816dd2e
I20260618 23:32:04.556713 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:04.550707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56937 (local address 127.0.98.62:40723)
0618 23:32:04.550752 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:04.550754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:04.551124 (+ 370us) server_negotiation.cc:407] Connection header received
0618 23:32:04.551180 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:04.551180 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:04.551189 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:04.551259 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:04.551671 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.552172 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.552910 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.553114 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.553184 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:04.554060 (+ 876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:04.554061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:04.554062 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:04.554068 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:04.555333 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:04.555763 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:04.555764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:04.555765 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:04.555817 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:04.556038 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:04.556040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:04.556040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:04.556190 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:04.556268 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:04.556476 (+ 208us) server_negotiation.cc:299] Negotiation successful
0618 23:32:04.556598 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260618 23:32:04.582540 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40723
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:32:04.675920 8261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:04.676167 8261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:04.676192 8261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:04.677511 8261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:04.677561 8261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:04.677584 8261 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 23:32:04.677603 8261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:32:04.679472 8261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40723
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:04.679811 8261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:04.680086 8261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:04.685690 8269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:04.685706 8267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:04.685750 8266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:04.686131 8261 server_base.cc:1034] running on GCE node
I20260618 23:32:04.686412 8261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:04.686719 8261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:04.687903 8261 hybrid_clock.cc:648] HybridClock initialized: now 1781825524687866 us; error 54 us; skew 500 ppm
Jun 18 23:32:04 dist-test-slave-6224 krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825524, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:32:04.691026 8261 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260618 23:32:04.691709 8261 webserver.cc:460] Webserver started at http://127.0.98.3:43999/ using document root <none> and password file <none>
I20260618 23:32:04.691989 8261 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:04.692041 8261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:04.692237 8261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:32:04.693470 8261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "bd221a0a77b140149fa64959904c1af4"
format_stamp: "Formatted at 2026-06-18 23:32:04 on dist-test-slave-6224"
server_key: "a6fd50b8fdb05d0198cfa61fad8e35a0"
server_key_iv: "d261bc7d14d849f5dd545486bf2ecdfb"
server_key_version: "encryptionkey@0"
I20260618 23:32:04.693885 8261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bd221a0a77b140149fa64959904c1af4"
format_stamp: "Formatted at 2026-06-18 23:32:04 on dist-test-slave-6224"
server_key: "a6fd50b8fdb05d0198cfa61fad8e35a0"
server_key_iv: "d261bc7d14d849f5dd545486bf2ecdfb"
server_key_version: "encryptionkey@0"
I20260618 23:32:04.695638 8261 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260618 23:32:04.696688 8276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:04.696983 8261 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:32:04.697130 8261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "bd221a0a77b140149fa64959904c1af4"
format_stamp: "Formatted at 2026-06-18 23:32:04 on dist-test-slave-6224"
server_key: "a6fd50b8fdb05d0198cfa61fad8e35a0"
server_key_iv: "d261bc7d14d849f5dd545486bf2ecdfb"
server_key_version: "encryptionkey@0"
I20260618 23:32:04.697233 8261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:04.712874 8261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:04.715319 8261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:04.715503 8261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:04.716234 8261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:32:04.716303 8261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:04.716348 8261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:32:04.716396 8261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 23:32:04 dist-test-slave-6224 krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825524, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260618 23:32:04.725844 8261 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43095
I20260618 23:32:04.725934 8391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43095 every 8 connection(s)
I20260618 23:32:04.726392 8261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:32:04.728484 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 8261
I20260618 23:32:04.728602 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:32:04.728626 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:04.716892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54541 (local address 127.0.98.62:40723)
0618 23:32:04.716940 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:04.716942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:04.717274 (+ 332us) server_negotiation.cc:407] Connection header received
0618 23:32:04.717825 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:04.717827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:04.717846 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:04.717945 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:04.719013 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.720037 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.721175 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.721404 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.721501 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:04.725252 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:04.725254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:04.725254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:04.725263 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:04.727086 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:04.727508 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:04.727509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:04.727509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:04.727626 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:04.727891 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:04.727892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:04.727892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:04.728021 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:04.728078 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:04.728397 (+ 319us) server_negotiation.cc:299] Negotiation successful
0618 23:32:04.728501 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260618 23:32:04.728674 8285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:04.716901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40723 (local address 127.0.98.3:54541)
0618 23:32:04.717115 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:04.717158 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:04.717699 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:04.718082 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:04.718086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:04.718513 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:04.718849 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:04.718856 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.720166 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.720167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:04.721058 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:04.721060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.721511 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.721512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:04.721645 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:04.722315 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:04.722330 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:04.725143 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:04.727177 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:04.727182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:04.727183 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:04.727419 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:04.727702 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:04.727703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:04.727703 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:04.727814 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:04.728086 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:04.728090 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:04.728215 (+ 125us) client_negotiation.cc:769] Sending connection context
0618 23:32:04.728308 (+ 93us) client_negotiation.cc:240] Negotiation successful
0618 23:32:04.728440 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":116,"threads_started":1}
I20260618 23:32:04.728840 392 external_mini_cluster.cc:1428] Setting key 8cd77a92d79a772bb2e58c3587a41f8a
I20260618 23:32:04.729516 8392 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40723
I20260618 23:32:04.729629 8392 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:04.729858 8392 heartbeater.cc:511] Master 127.0.98.62:40723 requested a full tablet report, sending...
I20260618 23:32:04.730351 7909 ts_manager.cc:194] Registered new tserver with Master: bd221a0a77b140149fa64959904c1af4 (127.0.98.3:43095)
I20260618 23:32:04.730885 7909 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54541
I20260618 23:32:04.736632 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:04.730813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47833 (local address 127.0.98.62:40723)
0618 23:32:04.730864 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:04.730866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:04.731039 (+ 173us) server_negotiation.cc:407] Connection header received
0618 23:32:04.731140 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:04.731141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:04.731152 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:04.731206 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:04.731609 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.732221 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.732975 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.733185 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.733270 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:04.734199 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:04.734200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:04.734201 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:04.734208 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:04.735436 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:04.735843 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:04.735844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:04.735845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:04.735902 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:04.736100 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:04.736101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:04.736101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:04.736218 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:04.736303 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:04.736448 (+ 145us) server_negotiation.cc:299] Negotiation successful
0618 23:32:04.736530 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260618 23:32:04.736645 8398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:04.730713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40723 (local address 127.0.98.3:47833)
0618 23:32:04.730951 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:04.730961 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:04.731070 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:04.731210 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:04.731211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:04.731431 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:04.731500 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:04.731501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.732315 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.732316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:04.732887 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:04.732889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.733271 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.733272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:04.733353 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:04.733780 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:04.733784 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:04.734110 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:04.735535 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:04.735536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:04.735537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:04.735750 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:04.735968 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:04.735969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:04.735969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:04.736038 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:04.736286 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:04.736287 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:04.736334 (+ 47us) client_negotiation.cc:769] Sending connection context
0618 23:32:04.736425 (+ 91us) client_negotiation.cc:240] Negotiation successful
0618 23:32:04.736541 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":121,"threads_started":1}
I20260618 23:32:04.741070 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 23:32:04 dist-test-slave-6224 krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825524, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 23:32:04 dist-test-slave-6224 krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:04 dist-test-slave-6224 krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260618 23:32:04.765185 7959 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:04.758628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59600 (local address 127.0.98.62:40723)
0618 23:32:04.758713 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:04.758716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:04.758727 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:32:04.758777 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:04.758778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:04.758789 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:04.758867 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:04.759621 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.760208 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.761056 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.761273 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.761390 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:04.765042 (+ 3652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59600: BlockingRecv error: recv got EOF from 127.0.0.1:59600 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 18 23:32:04 dist-test-slave-6224 krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825524, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260618 23:32:04.773711 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:04.766220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59604 (local address 127.0.98.62:40723)
0618 23:32:04.766290 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:04.766292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:04.766302 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:32:04.766353 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:04.766354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:04.766371 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:04.766437 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:04.766939 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.767443 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.768218 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.768410 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.768483 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:04.770953 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:04.770955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:04.770955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:04.770981 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:04.772530 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:04.772940 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:04.772941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:04.772941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:04.772997 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:04.773186 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:04.773187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:04.773187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:04.773299 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:04.773384 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:04.773576 (+ 192us) server_negotiation.cc:299] Negotiation successful
0618 23:32:04.773640 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 23:32:04.775820 7909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:32:04.777527 7909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:32:04.785038 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:04.780576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36853 (local address 127.0.0.1:49322)
0618 23:32:04.780770 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:04.780818 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:04.780902 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:04.781323 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:04.781326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:04.781329 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:04.781423 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:04.781426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.782456 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.782457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:04.783654 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:04.783656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.784644 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.784645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:04.784750 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:04.784768 (+ 18us) client_negotiation.cc:769] Sending connection context
0618 23:32:04.784858 (+ 90us) client_negotiation.cc:240] Negotiation successful
0618 23:32:04.784938 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":64,"threads_started":1}
I20260618 23:32:04.785050 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:04.780689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49322 (local address 127.0.98.1:36853)
0618 23:32:04.780880 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:04.780886 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:04.780900 (+ 14us) server_negotiation.cc:407] Connection header received
0618 23:32:04.781072 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:04.781077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:04.781098 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:04.781181 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:04.781516 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.782376 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.783754 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.784545 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.784626 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:04.784744 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:04.784887 (+ 143us) server_negotiation.cc:299] Negotiation successful
0618 23:32:04.784928 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":120,"threads_started":1}
I20260618 23:32:04.785338 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:04.780833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40739 (local address 127.0.0.1:34690)
0618 23:32:04.781357 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:04.781367 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:04.781419 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:04.781669 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:04.781670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:04.781672 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:04.781717 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:04.781718 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.782946 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.782947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:04.784080 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:04.784082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.785071 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.785072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:04.785133 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:04.785138 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:32:04.785191 (+ 53us) client_negotiation.cc:240] Negotiation successful
0618 23:32:04.785265 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":84,"threads_started":1}
I20260618 23:32:04.785604 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:04.781073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34690 (local address 127.0.98.2:40739)
0618 23:32:04.781314 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:04.781319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:04.781492 (+ 173us) server_negotiation.cc:407] Connection header received
0618 23:32:04.781542 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:04.781546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:04.781561 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:04.781647 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:04.781819 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.782848 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.784171 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.784834 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.784960 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:04.785045 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:04.785449 (+ 404us) server_negotiation.cc:299] Negotiation successful
0618 23:32:04.785496 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":121,"threads_started":1}
I20260618 23:32:04.785969 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:04.780580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43095 (local address 127.0.0.1:38958)
0618 23:32:04.780977 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:04.780990 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:04.781050 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:04.781387 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:04.781388 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:04.781390 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:04.781474 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:04.781476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.782799 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.782800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:04.784215 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:04.784217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.785717 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.785718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:04.785785 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:04.785790 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:32:04.785850 (+ 60us) client_negotiation.cc:240] Negotiation successful
0618 23:32:04.785898 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":54,"thread_start_us":126,"threads_started":1}
I20260618 23:32:04.786197 8046 tablet_service.cc:1467] Processing CreateTablet for tablet a45ad5202d0040f192324a4ff74a9914 (DEFAULT_TABLE table=test-table [id=bd7e8c59e5af4b5599cacb285163107f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:32:04.786194 8186 tablet_service.cc:1467] Processing CreateTablet for tablet a45ad5202d0040f192324a4ff74a9914 (DEFAULT_TABLE table=test-table [id=bd7e8c59e5af4b5599cacb285163107f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:32:04.786610 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a45ad5202d0040f192324a4ff74a9914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:04.786610 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a45ad5202d0040f192324a4ff74a9914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:04.787755 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:04.780919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38958 (local address 127.0.98.3:43095)
0618 23:32:04.781131 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:04.781134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:04.781144 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:32:04.781185 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:04.781187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:04.781203 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:04.781278 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:04.781554 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.782702 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.784316 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:04.785041 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:04.785089 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:04.785652 (+ 563us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:04.787635 (+ 1983us) server_negotiation.cc:299] Negotiation successful
0618 23:32:04.787678 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":106,"threads_started":1}
I20260618 23:32:04.788335 8326 tablet_service.cc:1467] Processing CreateTablet for tablet a45ad5202d0040f192324a4ff74a9914 (DEFAULT_TABLE table=test-table [id=bd7e8c59e5af4b5599cacb285163107f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:32:04.788630 8326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a45ad5202d0040f192324a4ff74a9914. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:04.791294 8421 tablet_bootstrap.cc:492] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f: Bootstrap starting.
I20260618 23:32:04.791576 8422 tablet_bootstrap.cc:492] T a45ad5202d0040f192324a4ff74a9914 P bd221a0a77b140149fa64959904c1af4: Bootstrap starting.
I20260618 23:32:04.792034 8421 tablet_bootstrap.cc:654] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:04.792330 8422 tablet_bootstrap.cc:654] T a45ad5202d0040f192324a4ff74a9914 P bd221a0a77b140149fa64959904c1af4: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:04.792372 8421 log.cc:824] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f: Log is configured to *not* fsync() on all Append() calls
I20260618 23:32:04.792662 8422 log.cc:824] T a45ad5202d0040f192324a4ff74a9914 P bd221a0a77b140149fa64959904c1af4: Log is configured to *not* fsync() on all Append() calls
I20260618 23:32:04.792757 8420 tablet_bootstrap.cc:492] T a45ad5202d0040f192324a4ff74a9914 P e92c821b3841414c965660b54194112f: Bootstrap starting.
I20260618 23:32:04.793213 8421 tablet_bootstrap.cc:492] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f: No bootstrap required, opened a new log
I20260618 23:32:04.793304 8421 ts_tablet_manager.cc:1397] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:32:04.793421 8420 tablet_bootstrap.cc:654] T a45ad5202d0040f192324a4ff74a9914 P e92c821b3841414c965660b54194112f: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:04.793541 8422 tablet_bootstrap.cc:492] T a45ad5202d0040f192324a4ff74a9914 P bd221a0a77b140149fa64959904c1af4: No bootstrap required, opened a new log
I20260618 23:32:04.793613 8422 ts_tablet_manager.cc:1397] T a45ad5202d0040f192324a4ff74a9914 P bd221a0a77b140149fa64959904c1af4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:32:04.793740 8420 log.cc:824] T a45ad5202d0040f192324a4ff74a9914 P e92c821b3841414c965660b54194112f: Log is configured to *not* fsync() on all Append() calls
I20260618 23:32:04.794468 8420 tablet_bootstrap.cc:492] T a45ad5202d0040f192324a4ff74a9914 P e92c821b3841414c965660b54194112f: No bootstrap required, opened a new log
I20260618 23:32:04.794554 8420 ts_tablet_manager.cc:1397] T a45ad5202d0040f192324a4ff74a9914 P e92c821b3841414c965660b54194112f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:32:04.795670 8421 raft_consensus.cc:348] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } }
I20260618 23:32:04.795783 8422 raft_consensus.cc:348] T a45ad5202d0040f192324a4ff74a9914 P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } }
I20260618 23:32:04.795884 8421 raft_consensus.cc:374] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:04.795914 8422 raft_consensus.cc:374] T a45ad5202d0040f192324a4ff74a9914 P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:04.795924 8421 raft_consensus.cc:729] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b9016f98a5411bb034d5e44b4dd52f, State: Initialized, Role: FOLLOWER
I20260618 23:32:04.795958 8422 raft_consensus.cc:729] T a45ad5202d0040f192324a4ff74a9914 P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd221a0a77b140149fa64959904c1af4, State: Initialized, Role: FOLLOWER
I20260618 23:32:04.796057 8421 consensus_queue.cc:260] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } }
I20260618 23:32:04.796082 8422 consensus_queue.cc:260] T a45ad5202d0040f192324a4ff74a9914 P bd221a0a77b140149fa64959904c1af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } }
I20260618 23:32:04.796383 8422 ts_tablet_manager.cc:1428] T a45ad5202d0040f192324a4ff74a9914 P bd221a0a77b140149fa64959904c1af4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:32:04.796383 8421 ts_tablet_manager.cc:1428] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:32:04.796257 8420 raft_consensus.cc:348] T a45ad5202d0040f192324a4ff74a9914 P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } }
I20260618 23:32:04.796448 8420 raft_consensus.cc:374] T a45ad5202d0040f192324a4ff74a9914 P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:04.796435 8112 heartbeater.cc:503] Master 127.0.98.62:40723 was elected leader, sending a full tablet report...
I20260618 23:32:04.796481 8420 raft_consensus.cc:729] T a45ad5202d0040f192324a4ff74a9914 P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92c821b3841414c965660b54194112f, State: Initialized, Role: FOLLOWER
I20260618 23:32:04.796530 8392 heartbeater.cc:503] Master 127.0.98.62:40723 was elected leader, sending a full tablet report...
I20260618 23:32:04.796604 8420 consensus_queue.cc:260] T a45ad5202d0040f192324a4ff74a9914 P e92c821b3841414c965660b54194112f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } }
I20260618 23:32:04.796883 8420 ts_tablet_manager.cc:1428] T a45ad5202d0040f192324a4ff74a9914 P e92c821b3841414c965660b54194112f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260618 23:32:04.797025 8252 heartbeater.cc:503] Master 127.0.98.62:40723 was elected leader, sending a full tablet report...
W20260618 23:32:04.881462 8113 tablet.cc:2369] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:32:04.977102 8393 tablet.cc:2369] T a45ad5202d0040f192324a4ff74a9914 P bd221a0a77b140149fa64959904c1af4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:32:05.034911 8426 raft_consensus.cc:482] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:32:05.035101 8426 raft_consensus.cc:504] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } }
I20260618 23:32:05.035513 8426 leader_election.cc:288] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd221a0a77b140149fa64959904c1af4 (127.0.98.3:43095), e92c821b3841414c965660b54194112f (127.0.98.2:40739)
I20260618 23:32:05.038913 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.035936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36759 (local address 127.0.98.3:43095)
0618 23:32:05.036159 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:05.036162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:05.036184 (+ 22us) server_negotiation.cc:407] Connection header received
0618 23:32:05.036301 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:05.036302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:05.036316 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:05.036394 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:05.036576 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.037098 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.038175 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.038541 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.038625 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.038639 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:05.038791 (+ 152us) server_negotiation.cc:299] Negotiation successful
0618 23:32:05.038827 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260618 23:32:05.038970 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.035779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43095 (local address 127.0.98.1:36759)
0618 23:32:05.035958 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:05.036093 (+ 135us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:05.036206 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:05.036399 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:05.036400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:05.036403 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:05.036501 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.036504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.037198 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.037199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.038070 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.038072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.038632 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.038633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.038710 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.038714 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:32:05.038784 (+ 70us) client_negotiation.cc:240] Negotiation successful
0618 23:32:05.038836 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":95,"threads_started":1}
I20260618 23:32:05.038959 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.036091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50595 (local address 127.0.98.2:40739)
0618 23:32:05.036160 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:05.036162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:05.036238 (+ 76us) server_negotiation.cc:407] Connection header received
0618 23:32:05.036299 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:05.036300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:05.036312 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:05.036393 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:05.036568 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.037105 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.038228 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.038590 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.038678 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.038689 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:05.038846 (+ 157us) server_negotiation.cc:299] Negotiation successful
0618 23:32:05.038881 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 23:32:05.039206 8430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.035992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40739 (local address 127.0.98.1:50595)
0618 23:32:05.036139 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:05.036154 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:05.036234 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:05.036398 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:05.036399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:05.036400 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:05.036480 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.036482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.037221 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.037222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.038131 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.038133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.038680 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.038681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.038758 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.038764 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:32:05.038829 (+ 65us) client_negotiation.cc:240] Negotiation successful
0618 23:32:05.038883 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":43,"thread_start_us":69,"threads_started":1}
I20260618 23:32:05.039363 8346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45ad5202d0040f192324a4ff74a9914" candidate_uuid: "20b9016f98a5411bb034d5e44b4dd52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd221a0a77b140149fa64959904c1af4" is_pre_election: true
I20260618 23:32:05.039546 8346 raft_consensus.cc:2454] T a45ad5202d0040f192324a4ff74a9914 P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b9016f98a5411bb034d5e44b4dd52f in term 0.
I20260618 23:32:05.039789 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45ad5202d0040f192324a4ff74a9914" candidate_uuid: "20b9016f98a5411bb034d5e44b4dd52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92c821b3841414c965660b54194112f" is_pre_election: true
I20260618 23:32:05.039855 7998 leader_election.cc:302] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b9016f98a5411bb034d5e44b4dd52f, bd221a0a77b140149fa64959904c1af4; no voters:
I20260618 23:32:05.039995 8206 raft_consensus.cc:2454] T a45ad5202d0040f192324a4ff74a9914 P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b9016f98a5411bb034d5e44b4dd52f in term 0.
I20260618 23:32:05.040050 8426 raft_consensus.cc:2781] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:32:05.040163 8426 raft_consensus.cc:482] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:32:05.040208 8426 raft_consensus.cc:3037] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:05.040980 8426 raft_consensus.cc:504] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } }
I20260618 23:32:05.041131 8426 leader_election.cc:288] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f [CANDIDATE]: Term 1 election: Requested vote from peers bd221a0a77b140149fa64959904c1af4 (127.0.98.3:43095), e92c821b3841414c965660b54194112f (127.0.98.2:40739)
I20260618 23:32:05.041352 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45ad5202d0040f192324a4ff74a9914" candidate_uuid: "20b9016f98a5411bb034d5e44b4dd52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92c821b3841414c965660b54194112f"
I20260618 23:32:05.041356 8346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a45ad5202d0040f192324a4ff74a9914" candidate_uuid: "20b9016f98a5411bb034d5e44b4dd52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd221a0a77b140149fa64959904c1af4"
I20260618 23:32:05.041466 8206 raft_consensus.cc:3037] T a45ad5202d0040f192324a4ff74a9914 P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:05.041458 8346 raft_consensus.cc:3037] T a45ad5202d0040f192324a4ff74a9914 P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:05.042191 8346 raft_consensus.cc:2454] T a45ad5202d0040f192324a4ff74a9914 P bd221a0a77b140149fa64959904c1af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b9016f98a5411bb034d5e44b4dd52f in term 1.
I20260618 23:32:05.042379 7998 leader_election.cc:302] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b9016f98a5411bb034d5e44b4dd52f, bd221a0a77b140149fa64959904c1af4; no voters:
I20260618 23:32:05.042449 8206 raft_consensus.cc:2454] T a45ad5202d0040f192324a4ff74a9914 P e92c821b3841414c965660b54194112f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b9016f98a5411bb034d5e44b4dd52f in term 1.
I20260618 23:32:05.042537 8426 raft_consensus.cc:2781] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:32:05.042694 8426 raft_consensus.cc:686] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f [term 1 LEADER]: Becoming Leader. State: Replica: 20b9016f98a5411bb034d5e44b4dd52f, State: Running, Role: LEADER
I20260618 23:32:05.042812 8426 consensus_queue.cc:237] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } }
I20260618 23:32:05.043779 7908 catalog_manager.cc:5495] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f reported cstate change: term changed from 0 to 1, leader changed from <none> to 20b9016f98a5411bb034d5e44b4dd52f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "20b9016f98a5411bb034d5e44b4dd52f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } health_report { overall_health: HEALTHY } } }
W20260618 23:32:05.047163 8253 tablet.cc:2369] T a45ad5202d0040f192324a4ff74a9914 P e92c821b3841414c965660b54194112f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:32:05.076412 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.072626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49334 (local address 127.0.98.1:36853)
0618 23:32:05.072714 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:05.072717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:05.072727 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:32:05.072774 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:05.072775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:05.072802 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:05.072951 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:32:05.073091 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.073754 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.075198 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.075604 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.075718 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.075972 (+ 254us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:32:05.076153 (+ 181us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:32:05.076249 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:05.076284 (+ 35us) server_negotiation.cc:299] Negotiation successful
0618 23:32:05.076310 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 23:32:05.079059 8206 raft_consensus.cc:1261] T a45ad5202d0040f192324a4ff74a9914 P e92c821b3841414c965660b54194112f [term 1 FOLLOWER]: Refusing update from remote peer 20b9016f98a5411bb034d5e44b4dd52f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:32:05.079154 8346 raft_consensus.cc:1261] T a45ad5202d0040f192324a4ff74a9914 P bd221a0a77b140149fa64959904c1af4 [term 1 FOLLOWER]: Refusing update from remote peer 20b9016f98a5411bb034d5e44b4dd52f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:32:05.079495 8426 consensus_queue.cc:1035] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:32:05.079828 8426 consensus_queue.cc:1035] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:32:05.082414 8434 mvcc.cc:205] Tried to move back new op lower bound from 7298357350720380928 to 7298357350575996928. Current Snapshot: MvccSnapshot[applied={T|T < 7298357350720380928}]
I20260618 23:32:05.085633 7909 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:32:05.085801 7909 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:32:05.086679 7909 catalog_manager.cc:5782] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f: Sending DeleteTablet for 3 replicas of tablet a45ad5202d0040f192324a4ff74a9914
I20260618 23:32:05.087049 8046 tablet_service.cc:1514] Processing DeleteTablet for tablet a45ad5202d0040f192324a4ff74a9914 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:32:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49322
I20260618 23:32:05.087049 8326 tablet_service.cc:1514] Processing DeleteTablet for tablet a45ad5202d0040f192324a4ff74a9914 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:32:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38958
I20260618 23:32:05.087201 7909 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59604:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:32:05.087349 7909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:32:05.087344 8445 tablet_replica.cc:330] T a45ad5202d0040f192324a4ff74a9914 P bd221a0a77b140149fa64959904c1af4: stopping tablet replica
I20260618 23:32:05.087364 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet a45ad5202d0040f192324a4ff74a9914 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:32:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34690
I20260618 23:32:05.087459 8445 raft_consensus.cc:2229] T a45ad5202d0040f192324a4ff74a9914 P bd221a0a77b140149fa64959904c1af4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:32:05.087594 8445 raft_consensus.cc:2258] T a45ad5202d0040f192324a4ff74a9914 P bd221a0a77b140149fa64959904c1af4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:32:05.087994 8446 tablet_replica.cc:330] T a45ad5202d0040f192324a4ff74a9914 P e92c821b3841414c965660b54194112f: stopping tablet replica
I20260618 23:32:05.088167 8446 raft_consensus.cc:2229] T a45ad5202d0040f192324a4ff74a9914 P e92c821b3841414c965660b54194112f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:32:05.088199 8445 ts_tablet_manager.cc:1905] T a45ad5202d0040f192324a4ff74a9914 P bd221a0a77b140149fa64959904c1af4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:32:05.088368 8446 raft_consensus.cc:2258] T a45ad5202d0040f192324a4ff74a9914 P e92c821b3841414c965660b54194112f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:32:05.088629 8444 tablet_replica.cc:330] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f: stopping tablet replica
I20260618 23:32:05.088806 8444 raft_consensus.cc:2229] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:32:05.088979 8444 raft_consensus.cc:2258] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:32:05.088963 8046 tablet_service.cc:1467] Processing CreateTablet for tablet dfcee1f2bd5f41829f2352cf930e3b5b (DEFAULT_TABLE table=test-table [id=7e2d7b4810f943f79e9715ed850bc6c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:32:05.089064 8186 tablet_service.cc:1467] Processing CreateTablet for tablet dfcee1f2bd5f41829f2352cf930e3b5b (DEFAULT_TABLE table=test-table [id=7e2d7b4810f943f79e9715ed850bc6c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:32:05.089152 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfcee1f2bd5f41829f2352cf930e3b5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:05.089160 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfcee1f2bd5f41829f2352cf930e3b5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:05.090422 8445 ts_tablet_manager.cc:1918] T a45ad5202d0040f192324a4ff74a9914 P bd221a0a77b140149fa64959904c1af4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:32:05.090488 8445 log.cc:1196] T a45ad5202d0040f192324a4ff74a9914 P bd221a0a77b140149fa64959904c1af4: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/a45ad5202d0040f192324a4ff74a9914
I20260618 23:32:05.090700 8421 tablet_bootstrap.cc:492] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f: Bootstrap starting.
I20260618 23:32:05.090760 8445 ts_tablet_manager.cc:1939] T a45ad5202d0040f192324a4ff74a9914 P bd221a0a77b140149fa64959904c1af4: Deleting consensus metadata
I20260618 23:32:05.090785 8326 tablet_service.cc:1467] Processing CreateTablet for tablet dfcee1f2bd5f41829f2352cf930e3b5b (DEFAULT_TABLE table=test-table [id=7e2d7b4810f943f79e9715ed850bc6c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:32:05.090912 8326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfcee1f2bd5f41829f2352cf930e3b5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:05.091174 8421 tablet_bootstrap.cc:654] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:05.092381 8444 ts_tablet_manager.cc:1905] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:32:05.092618 8420 tablet_bootstrap.cc:492] T dfcee1f2bd5f41829f2352cf930e3b5b P e92c821b3841414c965660b54194112f: Bootstrap starting.
I20260618 23:32:05.093084 8420 tablet_bootstrap.cc:654] T dfcee1f2bd5f41829f2352cf930e3b5b P e92c821b3841414c965660b54194112f: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:05.093447 8422 tablet_bootstrap.cc:492] T dfcee1f2bd5f41829f2352cf930e3b5b P bd221a0a77b140149fa64959904c1af4: Bootstrap starting.
I20260618 23:32:05.093926 8422 tablet_bootstrap.cc:654] T dfcee1f2bd5f41829f2352cf930e3b5b P bd221a0a77b140149fa64959904c1af4: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:05.094111 8421 tablet_bootstrap.cc:492] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f: No bootstrap required, opened a new log
I20260618 23:32:05.094157 8421 ts_tablet_manager.cc:1397] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260618 23:32:05.094362 8421 raft_consensus.cc:348] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.094466 8421 raft_consensus.cc:374] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:05.094493 8421 raft_consensus.cc:729] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b9016f98a5411bb034d5e44b4dd52f, State: Initialized, Role: FOLLOWER
I20260618 23:32:05.094547 8421 consensus_queue.cc:260] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.094666 8421 ts_tablet_manager.cc:1428] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260618 23:32:05.094669 7894 catalog_manager.cc:4841] TS bd221a0a77b140149fa64959904c1af4 (127.0.98.3:43095): tablet a45ad5202d0040f192324a4ff74a9914 (table test-table [id=bd7e8c59e5af4b5599cacb285163107f]) successfully deleted
I20260618 23:32:05.095608 8444 ts_tablet_manager.cc:1918] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:32:05.095745 8444 log.cc:1196] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-0/wal/wals/a45ad5202d0040f192324a4ff74a9914
I20260618 23:32:05.095769 8422 tablet_bootstrap.cc:492] T dfcee1f2bd5f41829f2352cf930e3b5b P bd221a0a77b140149fa64959904c1af4: No bootstrap required, opened a new log
I20260618 23:32:05.095818 8422 ts_tablet_manager.cc:1397] T dfcee1f2bd5f41829f2352cf930e3b5b P bd221a0a77b140149fa64959904c1af4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:32:05.096019 8422 raft_consensus.cc:348] T dfcee1f2bd5f41829f2352cf930e3b5b P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.096096 8422 raft_consensus.cc:374] T dfcee1f2bd5f41829f2352cf930e3b5b P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:05.096103 8444 ts_tablet_manager.cc:1939] T a45ad5202d0040f192324a4ff74a9914 P 20b9016f98a5411bb034d5e44b4dd52f: Deleting consensus metadata
I20260618 23:32:05.096122 8422 raft_consensus.cc:729] T dfcee1f2bd5f41829f2352cf930e3b5b P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd221a0a77b140149fa64959904c1af4, State: Initialized, Role: FOLLOWER
I20260618 23:32:05.096181 8422 consensus_queue.cc:260] T dfcee1f2bd5f41829f2352cf930e3b5b P bd221a0a77b140149fa64959904c1af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.096307 8422 ts_tablet_manager.cc:1428] T dfcee1f2bd5f41829f2352cf930e3b5b P bd221a0a77b140149fa64959904c1af4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:05.096634 7893 catalog_manager.cc:4841] TS 20b9016f98a5411bb034d5e44b4dd52f (127.0.98.1:36853): tablet a45ad5202d0040f192324a4ff74a9914 (table test-table [id=bd7e8c59e5af4b5599cacb285163107f]) successfully deleted
I20260618 23:32:05.097149 8420 tablet_bootstrap.cc:492] T dfcee1f2bd5f41829f2352cf930e3b5b P e92c821b3841414c965660b54194112f: No bootstrap required, opened a new log
I20260618 23:32:05.097219 8420 ts_tablet_manager.cc:1397] T dfcee1f2bd5f41829f2352cf930e3b5b P e92c821b3841414c965660b54194112f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260618 23:32:05.097401 8420 raft_consensus.cc:348] T dfcee1f2bd5f41829f2352cf930e3b5b P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.097489 8420 raft_consensus.cc:374] T dfcee1f2bd5f41829f2352cf930e3b5b P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:05.097514 8420 raft_consensus.cc:729] T dfcee1f2bd5f41829f2352cf930e3b5b P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92c821b3841414c965660b54194112f, State: Initialized, Role: FOLLOWER
I20260618 23:32:05.097555 8420 consensus_queue.cc:260] T dfcee1f2bd5f41829f2352cf930e3b5b P e92c821b3841414c965660b54194112f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.097669 8420 ts_tablet_manager.cc:1428] T dfcee1f2bd5f41829f2352cf930e3b5b P e92c821b3841414c965660b54194112f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:05.097788 8446 ts_tablet_manager.cc:1905] T a45ad5202d0040f192324a4ff74a9914 P e92c821b3841414c965660b54194112f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:32:05.099676 8446 ts_tablet_manager.cc:1918] T a45ad5202d0040f192324a4ff74a9914 P e92c821b3841414c965660b54194112f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:32:05.099792 8446 log.cc:1196] T a45ad5202d0040f192324a4ff74a9914 P e92c821b3841414c965660b54194112f: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/a45ad5202d0040f192324a4ff74a9914
I20260618 23:32:05.100402 8446 ts_tablet_manager.cc:1939] T a45ad5202d0040f192324a4ff74a9914 P e92c821b3841414c965660b54194112f: Deleting consensus metadata
I20260618 23:32:05.100869 7894 catalog_manager.cc:4841] TS e92c821b3841414c965660b54194112f (127.0.98.2:40739): tablet a45ad5202d0040f192324a4ff74a9914 (table test-table [id=bd7e8c59e5af4b5599cacb285163107f]) successfully deleted
I20260618 23:32:05.106873 8426 raft_consensus.cc:482] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:32:05.106978 8426 raft_consensus.cc:504] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.107219 8426 leader_election.cc:288] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e92c821b3841414c965660b54194112f (127.0.98.2:40739), bd221a0a77b140149fa64959904c1af4 (127.0.98.3:43095)
I20260618 23:32:05.107443 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfcee1f2bd5f41829f2352cf930e3b5b" candidate_uuid: "20b9016f98a5411bb034d5e44b4dd52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92c821b3841414c965660b54194112f" is_pre_election: true
I20260618 23:32:05.107574 8206 raft_consensus.cc:2454] T dfcee1f2bd5f41829f2352cf930e3b5b P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b9016f98a5411bb034d5e44b4dd52f in term 0.
I20260618 23:32:05.107522 8346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfcee1f2bd5f41829f2352cf930e3b5b" candidate_uuid: "20b9016f98a5411bb034d5e44b4dd52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd221a0a77b140149fa64959904c1af4" is_pre_election: true
I20260618 23:32:05.107647 8346 raft_consensus.cc:2454] T dfcee1f2bd5f41829f2352cf930e3b5b P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b9016f98a5411bb034d5e44b4dd52f in term 0.
I20260618 23:32:05.107851 7998 leader_election.cc:302] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b9016f98a5411bb034d5e44b4dd52f, bd221a0a77b140149fa64959904c1af4; no voters:
I20260618 23:32:05.107985 8426 raft_consensus.cc:2781] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:32:05.108075 8426 raft_consensus.cc:482] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:32:05.108111 8426 raft_consensus.cc:3037] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:05.108942 8426 raft_consensus.cc:504] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.109076 8426 leader_election.cc:288] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f [CANDIDATE]: Term 1 election: Requested vote from peers e92c821b3841414c965660b54194112f (127.0.98.2:40739), bd221a0a77b140149fa64959904c1af4 (127.0.98.3:43095)
I20260618 23:32:05.109261 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfcee1f2bd5f41829f2352cf930e3b5b" candidate_uuid: "20b9016f98a5411bb034d5e44b4dd52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92c821b3841414c965660b54194112f"
I20260618 23:32:05.109292 8346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfcee1f2bd5f41829f2352cf930e3b5b" candidate_uuid: "20b9016f98a5411bb034d5e44b4dd52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd221a0a77b140149fa64959904c1af4"
I20260618 23:32:05.109340 8206 raft_consensus.cc:3037] T dfcee1f2bd5f41829f2352cf930e3b5b P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:05.109354 8346 raft_consensus.cc:3037] T dfcee1f2bd5f41829f2352cf930e3b5b P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:05.110208 8206 raft_consensus.cc:2454] T dfcee1f2bd5f41829f2352cf930e3b5b P e92c821b3841414c965660b54194112f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b9016f98a5411bb034d5e44b4dd52f in term 1.
I20260618 23:32:05.110208 8346 raft_consensus.cc:2454] T dfcee1f2bd5f41829f2352cf930e3b5b P bd221a0a77b140149fa64959904c1af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b9016f98a5411bb034d5e44b4dd52f in term 1.
I20260618 23:32:05.110378 7998 leader_election.cc:302] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b9016f98a5411bb034d5e44b4dd52f, bd221a0a77b140149fa64959904c1af4; no voters:
I20260618 23:32:05.110595 8426 raft_consensus.cc:2781] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:32:05.110674 8426 raft_consensus.cc:686] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f [term 1 LEADER]: Becoming Leader. State: Replica: 20b9016f98a5411bb034d5e44b4dd52f, State: Running, Role: LEADER
I20260618 23:32:05.110755 8426 consensus_queue.cc:237] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.111518 7909 catalog_manager.cc:5495] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f reported cstate change: term changed from 0 to 1, leader changed from <none> to 20b9016f98a5411bb034d5e44b4dd52f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "20b9016f98a5411bb034d5e44b4dd52f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } health_report { overall_health: UNKNOWN } } }
Jun 18 23:32:05 dist-test-slave-6224 krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825524, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260618 23:32:05.153442 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.144603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59608 (local address 127.0.98.62:40723)
0618 23:32:05.144683 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:05.144685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:05.144775 (+ 90us) server_negotiation.cc:407] Connection header received
0618 23:32:05.144867 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:05.144868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:05.144882 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:05.144975 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:05.145501 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.146388 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.147149 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.147495 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.147600 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.150533 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:05.150535 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:05.150535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:05.150541 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:05.152041 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:05.152579 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:05.152579 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:05.152580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:05.152632 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:05.152866 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:05.152866 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:05.152867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:05.152977 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:05.153030 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:05.153246 (+ 216us) server_negotiation.cc:299] Negotiation successful
0618 23:32:05.153361 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":50}
I20260618 23:32:05.153666 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.144503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40723 (local address 127.0.0.1:59608)
0618 23:32:05.144668 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:05.144688 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:05.144764 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:05.144991 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:05.144991 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:05.145342 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:05.145408 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.145410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.146489 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.146489 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.147065 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.147066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.147702 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.147703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.147751 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.148421 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:05.148426 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:05.150431 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:05.152140 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:05.152150 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:05.152153 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:05.152487 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:05.152709 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:05.152710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:05.152711 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:05.152792 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:05.153052 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:05.153059 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:05.153127 (+ 68us) client_negotiation.cc:769] Sending connection context
0618 23:32:05.153220 (+ 93us) client_negotiation.cc:240] Negotiation successful
0618 23:32:05.153463 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":78,"thread_start_us":103,"threads_started":1}
I20260618 23:32:05.156350 7909 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59608:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260618 23:32:05.156647 7909 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:32:05.158391 8046 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb848efd08545fab9cd3cb801e23fee (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6a298165c4524eb78a0cc07ab0728527]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 23:32:05.158491 8326 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb848efd08545fab9cd3cb801e23fee (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6a298165c4524eb78a0cc07ab0728527]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 23:32:05.158530 8186 tablet_service.cc:1467] Processing CreateTablet for tablet 9eb848efd08545fab9cd3cb801e23fee (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6a298165c4524eb78a0cc07ab0728527]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 23:32:05.158608 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb848efd08545fab9cd3cb801e23fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:05.158633 8326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb848efd08545fab9cd3cb801e23fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:05.158732 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9eb848efd08545fab9cd3cb801e23fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:05.160596 8420 tablet_bootstrap.cc:492] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f: Bootstrap starting.
I20260618 23:32:05.161073 8420 tablet_bootstrap.cc:654] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:05.161441 8422 tablet_bootstrap.cc:492] T 9eb848efd08545fab9cd3cb801e23fee P bd221a0a77b140149fa64959904c1af4: Bootstrap starting.
I20260618 23:32:05.161837 8420 tablet_bootstrap.cc:492] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f: No bootstrap required, opened a new log
I20260618 23:32:05.161876 8422 tablet_bootstrap.cc:654] T 9eb848efd08545fab9cd3cb801e23fee P bd221a0a77b140149fa64959904c1af4: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:05.161909 8420 ts_tablet_manager.cc:1397] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:05.162134 8420 raft_consensus.cc:348] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.162215 8420 raft_consensus.cc:374] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:05.162237 8420 raft_consensus.cc:729] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92c821b3841414c965660b54194112f, State: Initialized, Role: FOLLOWER
I20260618 23:32:05.162287 8421 tablet_bootstrap.cc:492] T 9eb848efd08545fab9cd3cb801e23fee P 20b9016f98a5411bb034d5e44b4dd52f: Bootstrap starting.
I20260618 23:32:05.162289 8420 consensus_queue.cc:260] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.162387 8420 ts_tablet_manager.cc:1428] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260618 23:32:05.162557 8428 tablet_replica.cc:439] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } }
I20260618 23:32:05.162629 8428 tablet_replica.cc:442] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:32:05.162724 8421 tablet_bootstrap.cc:654] T 9eb848efd08545fab9cd3cb801e23fee P 20b9016f98a5411bb034d5e44b4dd52f: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:05.163915 8422 tablet_bootstrap.cc:492] T 9eb848efd08545fab9cd3cb801e23fee P bd221a0a77b140149fa64959904c1af4: No bootstrap required, opened a new log
I20260618 23:32:05.163990 8422 ts_tablet_manager.cc:1397] T 9eb848efd08545fab9cd3cb801e23fee P bd221a0a77b140149fa64959904c1af4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260618 23:32:05.164191 8422 raft_consensus.cc:348] T 9eb848efd08545fab9cd3cb801e23fee P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.164319 8422 raft_consensus.cc:374] T 9eb848efd08545fab9cd3cb801e23fee P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:05.164357 8422 raft_consensus.cc:729] T 9eb848efd08545fab9cd3cb801e23fee P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd221a0a77b140149fa64959904c1af4, State: Initialized, Role: FOLLOWER
I20260618 23:32:05.164422 8422 consensus_queue.cc:260] T 9eb848efd08545fab9cd3cb801e23fee P bd221a0a77b140149fa64959904c1af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.164579 8422 ts_tablet_manager.cc:1428] T 9eb848efd08545fab9cd3cb801e23fee P bd221a0a77b140149fa64959904c1af4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:05.164862 8421 tablet_bootstrap.cc:492] T 9eb848efd08545fab9cd3cb801e23fee P 20b9016f98a5411bb034d5e44b4dd52f: No bootstrap required, opened a new log
I20260618 23:32:05.164927 8421 ts_tablet_manager.cc:1397] T 9eb848efd08545fab9cd3cb801e23fee P 20b9016f98a5411bb034d5e44b4dd52f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260618 23:32:05.165107 8421 raft_consensus.cc:348] T 9eb848efd08545fab9cd3cb801e23fee P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.165202 8421 raft_consensus.cc:374] T 9eb848efd08545fab9cd3cb801e23fee P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:05.165236 8421 raft_consensus.cc:729] T 9eb848efd08545fab9cd3cb801e23fee P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b9016f98a5411bb034d5e44b4dd52f, State: Initialized, Role: FOLLOWER
I20260618 23:32:05.165292 8421 consensus_queue.cc:260] T 9eb848efd08545fab9cd3cb801e23fee P 20b9016f98a5411bb034d5e44b4dd52f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.165396 8421 ts_tablet_manager.cc:1428] T 9eb848efd08545fab9cd3cb801e23fee P 20b9016f98a5411bb034d5e44b4dd52f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:05.165450 8439 tablet_replica.cc:439] T 9eb848efd08545fab9cd3cb801e23fee P 20b9016f98a5411bb034d5e44b4dd52f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } }
I20260618 23:32:05.165593 8439 tablet_replica.cc:442] T 9eb848efd08545fab9cd3cb801e23fee P 20b9016f98a5411bb034d5e44b4dd52f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:32:05.165604 8427 tablet_replica.cc:439] T 9eb848efd08545fab9cd3cb801e23fee P bd221a0a77b140149fa64959904c1af4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } }
I20260618 23:32:05.165720 8427 tablet_replica.cc:442] T 9eb848efd08545fab9cd3cb801e23fee P bd221a0a77b140149fa64959904c1af4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:32:05.260172 8428 raft_consensus.cc:482] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:32:05.260414 8428 raft_consensus.cc:504] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.260962 8428 leader_election.cc:288] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20b9016f98a5411bb034d5e44b4dd52f (127.0.98.1:36853), bd221a0a77b140149fa64959904c1af4 (127.0.98.3:43095)
I20260618 23:32:05.264393 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.261262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43207 (local address 127.0.98.3:43095)
0618 23:32:05.261327 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:05.261329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:05.261534 (+ 205us) server_negotiation.cc:407] Connection header received
0618 23:32:05.261659 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:05.261660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:05.261670 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:05.261742 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:05.261948 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.262449 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.263667 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.264032 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.264114 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.264126 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:05.264273 (+ 147us) server_negotiation.cc:299] Negotiation successful
0618 23:32:05.264308 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 23:32:05.264464 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.261262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59163 (local address 127.0.98.1:36853)
0618 23:32:05.261337 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:05.261340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:05.261530 (+ 190us) server_negotiation.cc:407] Connection header received
0618 23:32:05.261630 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:05.261631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:05.261642 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:05.261715 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:05.261908 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.262434 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.263664 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.264074 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.264167 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.264180 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:05.264335 (+ 155us) server_negotiation.cc:299] Negotiation successful
0618 23:32:05.264375 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 23:32:05.264530 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.261194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36853 (local address 127.0.98.2:59163)
0618 23:32:05.261433 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:05.261450 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:05.261544 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:05.261720 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:05.261721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:05.261729 (+ 8us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:05.261832 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.261835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.262523 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.262524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.263506 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.263508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.264170 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.264170 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.264253 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.264258 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:32:05.264329 (+ 71us) client_negotiation.cc:240] Negotiation successful
0618 23:32:05.264383 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":58,"thread_start_us":169,"threads_started":1}
I20260618 23:32:05.264416 8461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.261195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43095 (local address 127.0.98.2:43207)
0618 23:32:05.261450 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:05.261461 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:05.261565 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:05.261765 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:05.261766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:05.261771 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:05.261872 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.261875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.262536 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.262537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.263540 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.263543 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.264114 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.264116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.264197 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.264201 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:32:05.264268 (+ 67us) client_negotiation.cc:240] Negotiation successful
0618 23:32:05.264314 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":46,"spinlock_wait_cycles":1408,"thread_start_us":150,"threads_started":1}
I20260618 23:32:05.264824 8066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb848efd08545fab9cd3cb801e23fee" candidate_uuid: "e92c821b3841414c965660b54194112f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b9016f98a5411bb034d5e44b4dd52f" is_pre_election: true
I20260618 23:32:05.265017 8066 raft_consensus.cc:2454] T 9eb848efd08545fab9cd3cb801e23fee P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e92c821b3841414c965660b54194112f in term 0.
I20260618 23:32:05.265118 8346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb848efd08545fab9cd3cb801e23fee" candidate_uuid: "e92c821b3841414c965660b54194112f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd221a0a77b140149fa64959904c1af4" is_pre_election: true
I20260618 23:32:05.265246 8346 raft_consensus.cc:2454] T 9eb848efd08545fab9cd3cb801e23fee P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e92c821b3841414c965660b54194112f in term 0.
I20260618 23:32:05.265250 8137 leader_election.cc:302] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b9016f98a5411bb034d5e44b4dd52f, e92c821b3841414c965660b54194112f; no voters:
I20260618 23:32:05.265384 8428 raft_consensus.cc:2781] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:32:05.265431 8428 raft_consensus.cc:482] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:32:05.265456 8428 raft_consensus.cc:3037] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:05.266456 8428 raft_consensus.cc:504] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.266628 8428 leader_election.cc:288] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f [CANDIDATE]: Term 1 election: Requested vote from peers 20b9016f98a5411bb034d5e44b4dd52f (127.0.98.1:36853), bd221a0a77b140149fa64959904c1af4 (127.0.98.3:43095)
I20260618 23:32:05.266772 8066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb848efd08545fab9cd3cb801e23fee" candidate_uuid: "e92c821b3841414c965660b54194112f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b9016f98a5411bb034d5e44b4dd52f"
I20260618 23:32:05.266875 8066 raft_consensus.cc:3037] T 9eb848efd08545fab9cd3cb801e23fee P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:05.266950 8346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9eb848efd08545fab9cd3cb801e23fee" candidate_uuid: "e92c821b3841414c965660b54194112f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd221a0a77b140149fa64959904c1af4"
I20260618 23:32:05.267048 8346 raft_consensus.cc:3037] T 9eb848efd08545fab9cd3cb801e23fee P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:05.267803 8346 raft_consensus.cc:2454] T 9eb848efd08545fab9cd3cb801e23fee P bd221a0a77b140149fa64959904c1af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e92c821b3841414c965660b54194112f in term 1.
I20260618 23:32:05.267867 8066 raft_consensus.cc:2454] T 9eb848efd08545fab9cd3cb801e23fee P 20b9016f98a5411bb034d5e44b4dd52f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e92c821b3841414c965660b54194112f in term 1.
I20260618 23:32:05.268065 8137 leader_election.cc:302] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b9016f98a5411bb034d5e44b4dd52f, e92c821b3841414c965660b54194112f; no voters:
I20260618 23:32:05.268183 8428 raft_consensus.cc:2781] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:32:05.268388 8428 raft_consensus.cc:686] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f [term 1 LEADER]: Becoming Leader. State: Replica: e92c821b3841414c965660b54194112f, State: Running, Role: LEADER
I20260618 23:32:05.268491 8428 consensus_queue.cc:237] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.268954 8462 tablet_replica.cc:439] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f: TxnStatusTablet state changed. Reason: New leader e92c821b3841414c965660b54194112f. Latest consensus state: current_term: 1 leader_uuid: "e92c821b3841414c965660b54194112f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } }
I20260618 23:32:05.269085 8462 tablet_replica.cc:442] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f: This TxnStatusTablet replica's current role is: LEADER
I20260618 23:32:05.269377 8464 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260618 23:32:05.269517 8464 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260618 23:32:05.269436 7909 catalog_manager.cc:5495] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f reported cstate change: term changed from 0 to 1, leader changed from <none> to e92c821b3841414c965660b54194112f (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e92c821b3841414c965660b54194112f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } health_report { overall_health: UNKNOWN } } }
I20260618 23:32:05.284197 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.281276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40739 (local address 127.0.0.1:34694)
0618 23:32:05.281375 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:05.281390 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:05.281448 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:05.281662 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:05.281663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:05.281668 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:32:05.281734 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.281738 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.282561 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.282562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.283166 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.283167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.283584 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.283585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.283653 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.283672 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:32:05.284001 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:32:05.284011 (+ 10us) client_negotiation.cc:769] Sending connection context
0618 23:32:05.284067 (+ 56us) client_negotiation.cc:240] Negotiation successful
0618 23:32:05.284122 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260618 23:32:05.284195 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.281348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34694 (local address 127.0.98.2:40739)
0618 23:32:05.281411 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:05.281414 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:05.281426 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:32:05.281552 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:05.281553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:05.281575 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:05.281638 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:32:05.281827 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.282476 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.283253 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.283465 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.283597 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.283787 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:32:05.283932 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:32:05.284020 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:05.284073 (+ 53us) server_negotiation.cc:299] Negotiation successful
0618 23:32:05.284096 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260618 23:32:05.285637 8346 raft_consensus.cc:1261] T 9eb848efd08545fab9cd3cb801e23fee P bd221a0a77b140149fa64959904c1af4 [term 1 FOLLOWER]: Refusing update from remote peer e92c821b3841414c965660b54194112f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:32:05.285692 8066 raft_consensus.cc:1261] T 9eb848efd08545fab9cd3cb801e23fee P 20b9016f98a5411bb034d5e44b4dd52f [term 1 FOLLOWER]: Refusing update from remote peer e92c821b3841414c965660b54194112f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:32:05.286001 8428 consensus_queue.cc:1035] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:32:05.286273 8462 consensus_queue.cc:1035] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:32:05.286682 8427 tablet_replica.cc:439] T 9eb848efd08545fab9cd3cb801e23fee P bd221a0a77b140149fa64959904c1af4: TxnStatusTablet state changed. Reason: New leader e92c821b3841414c965660b54194112f. Latest consensus state: current_term: 1 leader_uuid: "e92c821b3841414c965660b54194112f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } }
I20260618 23:32:05.286778 8427 tablet_replica.cc:442] T 9eb848efd08545fab9cd3cb801e23fee P bd221a0a77b140149fa64959904c1af4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:32:05.287104 8439 tablet_replica.cc:439] T 9eb848efd08545fab9cd3cb801e23fee P 20b9016f98a5411bb034d5e44b4dd52f: TxnStatusTablet state changed. Reason: New leader e92c821b3841414c965660b54194112f. Latest consensus state: current_term: 1 leader_uuid: "e92c821b3841414c965660b54194112f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } }
I20260618 23:32:05.287212 8439 tablet_replica.cc:442] T 9eb848efd08545fab9cd3cb801e23fee P 20b9016f98a5411bb034d5e44b4dd52f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:32:05.287446 8428 tablet_replica.cc:439] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e92c821b3841414c965660b54194112f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } }
I20260618 23:32:05.287535 8428 tablet_replica.cc:442] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f: This TxnStatusTablet replica's current role is: LEADER
I20260618 23:32:05.287794 8436 mvcc.cc:205] Tried to move back new op lower bound from 7298357351567560704 to 7298357351500587008. Current Snapshot: MvccSnapshot[applied={T|T < 7298357351567560704}]
I20260618 23:32:05.287825 8435 mvcc.cc:205] Tried to move back new op lower bound from 7298357351567560704 to 7298357351500587008. Current Snapshot: MvccSnapshot[applied={T|T < 7298357351567560704 or (T in {7298357351567560704})}]
I20260618 23:32:05.287927 8427 tablet_replica.cc:439] T 9eb848efd08545fab9cd3cb801e23fee P bd221a0a77b140149fa64959904c1af4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e92c821b3841414c965660b54194112f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } }
I20260618 23:32:05.287997 8427 tablet_replica.cc:442] T 9eb848efd08545fab9cd3cb801e23fee P bd221a0a77b140149fa64959904c1af4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:32:05.288108 8462 tablet_replica.cc:439] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e92c821b3841414c965660b54194112f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } }
I20260618 23:32:05.288187 8462 tablet_replica.cc:442] T 9eb848efd08545fab9cd3cb801e23fee P e92c821b3841414c965660b54194112f: This TxnStatusTablet replica's current role is: LEADER
I20260618 23:32:05.288753 8439 tablet_replica.cc:439] T 9eb848efd08545fab9cd3cb801e23fee P 20b9016f98a5411bb034d5e44b4dd52f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e92c821b3841414c965660b54194112f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } } }
I20260618 23:32:05.288846 8439 tablet_replica.cc:442] T 9eb848efd08545fab9cd3cb801e23fee P 20b9016f98a5411bb034d5e44b4dd52f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:32:05.295444 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.292324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33627 (local address 127.0.98.2:40739)
0618 23:32:05.292435 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:05.292438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:05.292535 (+ 97us) server_negotiation.cc:407] Connection header received
0618 23:32:05.292607 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:05.292607 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:05.292618 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:05.292727 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:32:05.292894 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.293414 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.294624 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.294830 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.294902 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.295147 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:32:05.295187 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:32:05.295251 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:05.295347 (+ 96us) server_negotiation.cc:299] Negotiation successful
0618 23:32:05.295368 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 23:32:05.295492 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.292209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40739 (local address 127.0.98.1:33627)
0618 23:32:05.292438 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:05.292453 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:05.292531 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:05.292748 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:05.292749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:05.292751 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:32:05.292812 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.292814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.293495 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.293496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.294513 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.294515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.294924 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.294925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.294990 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.295007 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:32:05.295274 (+ 267us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:32:05.295276 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:32:05.295344 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 23:32:05.295395 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":130,"threads_started":1}
I20260618 23:32:05.298276 8346 raft_consensus.cc:1261] T dfcee1f2bd5f41829f2352cf930e3b5b P bd221a0a77b140149fa64959904c1af4 [term 1 FOLLOWER]: Refusing update from remote peer 20b9016f98a5411bb034d5e44b4dd52f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:32:05.298322 8206 raft_consensus.cc:1261] T dfcee1f2bd5f41829f2352cf930e3b5b P e92c821b3841414c965660b54194112f [term 1 FOLLOWER]: Refusing update from remote peer 20b9016f98a5411bb034d5e44b4dd52f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:32:05.298521 8439 consensus_queue.cc:1035] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:32:05.298641 8426 consensus_queue.cc:1035] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:32:05.308600 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.305431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59511 (local address 127.0.98.1:36853)
0618 23:32:05.305477 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:05.305480 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:05.305658 (+ 178us) server_negotiation.cc:407] Connection header received
0618 23:32:05.305746 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:05.305748 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:05.305762 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:05.305819 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:32:05.306002 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.306526 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.307783 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.307997 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.308082 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.308289 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:32:05.308329 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:32:05.308406 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:05.308470 (+ 64us) server_negotiation.cc:299] Negotiation successful
0618 23:32:05.308499 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:32:05.308677 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.305292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36853 (local address 127.0.98.2:59511)
0618 23:32:05.305564 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:05.305576 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:05.305675 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:05.305821 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:05.305822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:05.305824 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:32:05.305916 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.305919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.306611 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.306612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.307686 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.307689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.308082 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.308083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.308151 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.308179 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:32:05.308409 (+ 230us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:32:05.308411 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:32:05.308466 (+ 55us) client_negotiation.cc:240] Negotiation successful
0618 23:32:05.308525 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":118,"threads_started":1}
I20260618 23:32:05.317947 7908 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:32:05.318112 7908 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59604:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:32:05.318944 7908 catalog_manager.cc:5782] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f: Sending DeleteTablet for 3 replicas of tablet dfcee1f2bd5f41829f2352cf930e3b5b
I20260618 23:32:05.319276 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet dfcee1f2bd5f41829f2352cf930e3b5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:32:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34690
I20260618 23:32:05.319303 8326 tablet_service.cc:1514] Processing DeleteTablet for tablet dfcee1f2bd5f41829f2352cf930e3b5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:32:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38958
I20260618 23:32:05.319422 8446 tablet_replica.cc:330] T dfcee1f2bd5f41829f2352cf930e3b5b P e92c821b3841414c965660b54194112f: stopping tablet replica
I20260618 23:32:05.319423 8445 tablet_replica.cc:330] T dfcee1f2bd5f41829f2352cf930e3b5b P bd221a0a77b140149fa64959904c1af4: stopping tablet replica
I20260618 23:32:05.319484 8446 raft_consensus.cc:2229] T dfcee1f2bd5f41829f2352cf930e3b5b P e92c821b3841414c965660b54194112f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:32:05.319491 8445 raft_consensus.cc:2229] T dfcee1f2bd5f41829f2352cf930e3b5b P bd221a0a77b140149fa64959904c1af4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:32:05.319479 8046 tablet_service.cc:1514] Processing DeleteTablet for tablet dfcee1f2bd5f41829f2352cf930e3b5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:32:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49322
I20260618 23:32:05.319527 8446 raft_consensus.cc:2258] T dfcee1f2bd5f41829f2352cf930e3b5b P e92c821b3841414c965660b54194112f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:32:05.319535 8445 raft_consensus.cc:2258] T dfcee1f2bd5f41829f2352cf930e3b5b P bd221a0a77b140149fa64959904c1af4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:32:05.319613 8444 tablet_replica.cc:330] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f: stopping tablet replica
I20260618 23:32:05.319677 8444 raft_consensus.cc:2229] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:32:05.319757 8444 raft_consensus.cc:2258] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:32:05.320008 8446 ts_tablet_manager.cc:1905] T dfcee1f2bd5f41829f2352cf930e3b5b P e92c821b3841414c965660b54194112f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:32:05.320127 8445 ts_tablet_manager.cc:1905] T dfcee1f2bd5f41829f2352cf930e3b5b P bd221a0a77b140149fa64959904c1af4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:32:05.321940 8446 ts_tablet_manager.cc:1918] T dfcee1f2bd5f41829f2352cf930e3b5b P e92c821b3841414c965660b54194112f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:32:05.322031 8446 log.cc:1196] T dfcee1f2bd5f41829f2352cf930e3b5b P e92c821b3841414c965660b54194112f: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/dfcee1f2bd5f41829f2352cf930e3b5b
I20260618 23:32:05.322122 8445 ts_tablet_manager.cc:1918] T dfcee1f2bd5f41829f2352cf930e3b5b P bd221a0a77b140149fa64959904c1af4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:32:05.322196 8445 log.cc:1196] T dfcee1f2bd5f41829f2352cf930e3b5b P bd221a0a77b140149fa64959904c1af4: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/dfcee1f2bd5f41829f2352cf930e3b5b
I20260618 23:32:05.322279 8446 ts_tablet_manager.cc:1939] T dfcee1f2bd5f41829f2352cf930e3b5b P e92c821b3841414c965660b54194112f: Deleting consensus metadata
I20260618 23:32:05.322495 8445 ts_tablet_manager.cc:1939] T dfcee1f2bd5f41829f2352cf930e3b5b P bd221a0a77b140149fa64959904c1af4: Deleting consensus metadata
I20260618 23:32:05.322736 7894 catalog_manager.cc:4841] TS e92c821b3841414c965660b54194112f (127.0.98.2:40739): tablet dfcee1f2bd5f41829f2352cf930e3b5b (table test-table [id=7e2d7b4810f943f79e9715ed850bc6c7]) successfully deleted
I20260618 23:32:05.322865 7894 catalog_manager.cc:4841] TS bd221a0a77b140149fa64959904c1af4 (127.0.98.3:43095): tablet dfcee1f2bd5f41829f2352cf930e3b5b (table test-table [id=7e2d7b4810f943f79e9715ed850bc6c7]) successfully deleted
I20260618 23:32:05.323117 8444 ts_tablet_manager.cc:1905] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:32:05.324615 8444 ts_tablet_manager.cc:1918] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:32:05.324695 8444 log.cc:1196] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-0/wal/wals/dfcee1f2bd5f41829f2352cf930e3b5b
I20260618 23:32:05.324963 8444 ts_tablet_manager.cc:1939] T dfcee1f2bd5f41829f2352cf930e3b5b P 20b9016f98a5411bb034d5e44b4dd52f: Deleting consensus metadata
I20260618 23:32:05.325367 7893 catalog_manager.cc:4841] TS 20b9016f98a5411bb034d5e44b4dd52f (127.0.98.1:36853): tablet dfcee1f2bd5f41829f2352cf930e3b5b (table test-table [id=7e2d7b4810f943f79e9715ed850bc6c7]) successfully deleted
Jun 18 23:32:05 dist-test-slave-6224 krb5kdc[7864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 23:32:05 dist-test-slave-6224 krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:05 dist-test-slave-6224 krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260618 23:32:05.348960 7959 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:05.342284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59622 (local address 127.0.98.62:40723)
0618 23:32:05.342383 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:05.342386 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:05.342404 (+ 18us) server_negotiation.cc:407] Connection header received
0618 23:32:05.342473 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:05.342475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:05.342491 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:05.342596 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:05.343174 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.344048 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.344894 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.345210 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.345302 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.348878 (+ 3576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59622: BlockingRecv error: recv got EOF from 127.0.0.1:59622 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 18 23:32:05 dist-test-slave-6224 krb5kdc[7864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260618 23:32:05.358150 7959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.350161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59638 (local address 127.0.98.62:40723)
0618 23:32:05.350230 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:05.350232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:05.350242 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:32:05.350270 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:05.350271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:05.350288 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:05.350359 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:05.350804 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.351598 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.352313 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.352613 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.352692 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.354977 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:05.354979 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:05.354979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:05.354989 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:05.356946 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:05.357305 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:05.357306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:05.357307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:05.357376 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:05.357583 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:05.357584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:05.357585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:05.357738 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:05.357859 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:05.357952 (+ 93us) server_negotiation.cc:299] Negotiation successful
0618 23:32:05.358039 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 23:32:05.360112 7908 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:32:05.360348 7908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:32:05.362084 8186 tablet_service.cc:1467] Processing CreateTablet for tablet 009443acf2414655bb736bc7f7275afe (DEFAULT_TABLE table=test-table [id=364cc2e74c5149d5ab253b5c286d839d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:32:05.362125 8046 tablet_service.cc:1467] Processing CreateTablet for tablet 009443acf2414655bb736bc7f7275afe (DEFAULT_TABLE table=test-table [id=364cc2e74c5149d5ab253b5c286d839d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:32:05.362183 8326 tablet_service.cc:1467] Processing CreateTablet for tablet 009443acf2414655bb736bc7f7275afe (DEFAULT_TABLE table=test-table [id=364cc2e74c5149d5ab253b5c286d839d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:32:05.362238 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009443acf2414655bb736bc7f7275afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:05.362237 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009443acf2414655bb736bc7f7275afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:05.362360 8326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 009443acf2414655bb736bc7f7275afe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:05.363663 8421 tablet_bootstrap.cc:492] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f: Bootstrap starting.
I20260618 23:32:05.363993 8422 tablet_bootstrap.cc:492] T 009443acf2414655bb736bc7f7275afe P bd221a0a77b140149fa64959904c1af4: Bootstrap starting.
I20260618 23:32:05.364040 8421 tablet_bootstrap.cc:654] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:05.364305 8420 tablet_bootstrap.cc:492] T 009443acf2414655bb736bc7f7275afe P e92c821b3841414c965660b54194112f: Bootstrap starting.
I20260618 23:32:05.364440 8422 tablet_bootstrap.cc:654] T 009443acf2414655bb736bc7f7275afe P bd221a0a77b140149fa64959904c1af4: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:05.364640 8420 tablet_bootstrap.cc:654] T 009443acf2414655bb736bc7f7275afe P e92c821b3841414c965660b54194112f: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:05.365137 8422 tablet_bootstrap.cc:492] T 009443acf2414655bb736bc7f7275afe P bd221a0a77b140149fa64959904c1af4: No bootstrap required, opened a new log
I20260618 23:32:05.365204 8422 ts_tablet_manager.cc:1397] T 009443acf2414655bb736bc7f7275afe P bd221a0a77b140149fa64959904c1af4: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:05.365435 8420 tablet_bootstrap.cc:492] T 009443acf2414655bb736bc7f7275afe P e92c821b3841414c965660b54194112f: No bootstrap required, opened a new log
I20260618 23:32:05.365464 8421 tablet_bootstrap.cc:492] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f: No bootstrap required, opened a new log
I20260618 23:32:05.365420 8422 raft_consensus.cc:348] T 009443acf2414655bb736bc7f7275afe P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.365484 8420 ts_tablet_manager.cc:1397] T 009443acf2414655bb736bc7f7275afe P e92c821b3841414c965660b54194112f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:05.365496 8422 raft_consensus.cc:374] T 009443acf2414655bb736bc7f7275afe P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:05.365500 8421 ts_tablet_manager.cc:1397] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:32:05.365520 8422 raft_consensus.cc:729] T 009443acf2414655bb736bc7f7275afe P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd221a0a77b140149fa64959904c1af4, State: Initialized, Role: FOLLOWER
I20260618 23:32:05.365569 8422 consensus_queue.cc:260] T 009443acf2414655bb736bc7f7275afe P bd221a0a77b140149fa64959904c1af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.365613 8421 raft_consensus.cc:348] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.365659 8422 ts_tablet_manager.cc:1428] T 009443acf2414655bb736bc7f7275afe P bd221a0a77b140149fa64959904c1af4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260618 23:32:05.365646 8420 raft_consensus.cc:348] T 009443acf2414655bb736bc7f7275afe P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.365687 8421 raft_consensus.cc:374] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:05.365712 8420 raft_consensus.cc:374] T 009443acf2414655bb736bc7f7275afe P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:05.365713 8421 raft_consensus.cc:729] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b9016f98a5411bb034d5e44b4dd52f, State: Initialized, Role: FOLLOWER
I20260618 23:32:05.365733 8420 raft_consensus.cc:729] T 009443acf2414655bb736bc7f7275afe P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92c821b3841414c965660b54194112f, State: Initialized, Role: FOLLOWER
I20260618 23:32:05.365757 8421 consensus_queue.cc:260] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.365779 8420 consensus_queue.cc:260] T 009443acf2414655bb736bc7f7275afe P e92c821b3841414c965660b54194112f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.365852 8421 ts_tablet_manager.cc:1428] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:05.365882 8420 ts_tablet_manager.cc:1428] T 009443acf2414655bb736bc7f7275afe P e92c821b3841414c965660b54194112f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:05.394343 8426 raft_consensus.cc:482] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:32:05.394510 8426 raft_consensus.cc:504] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.394769 8426 leader_election.cc:288] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e92c821b3841414c965660b54194112f (127.0.98.2:40739), bd221a0a77b140149fa64959904c1af4 (127.0.98.3:43095)
I20260618 23:32:05.395015 8346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009443acf2414655bb736bc7f7275afe" candidate_uuid: "20b9016f98a5411bb034d5e44b4dd52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd221a0a77b140149fa64959904c1af4" is_pre_election: true
I20260618 23:32:05.395059 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009443acf2414655bb736bc7f7275afe" candidate_uuid: "20b9016f98a5411bb034d5e44b4dd52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92c821b3841414c965660b54194112f" is_pre_election: true
I20260618 23:32:05.395166 8206 raft_consensus.cc:2454] T 009443acf2414655bb736bc7f7275afe P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b9016f98a5411bb034d5e44b4dd52f in term 0.
I20260618 23:32:05.395166 8346 raft_consensus.cc:2454] T 009443acf2414655bb736bc7f7275afe P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20b9016f98a5411bb034d5e44b4dd52f in term 0.
I20260618 23:32:05.395345 7998 leader_election.cc:302] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b9016f98a5411bb034d5e44b4dd52f, e92c821b3841414c965660b54194112f; no voters:
I20260618 23:32:05.395493 8426 raft_consensus.cc:2781] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:32:05.395573 8426 raft_consensus.cc:482] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:32:05.395598 8426 raft_consensus.cc:3037] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:05.396739 8426 raft_consensus.cc:504] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.396905 8426 leader_election.cc:288] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f [CANDIDATE]: Term 1 election: Requested vote from peers e92c821b3841414c965660b54194112f (127.0.98.2:40739), bd221a0a77b140149fa64959904c1af4 (127.0.98.3:43095)
I20260618 23:32:05.397094 8346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009443acf2414655bb736bc7f7275afe" candidate_uuid: "20b9016f98a5411bb034d5e44b4dd52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd221a0a77b140149fa64959904c1af4"
I20260618 23:32:05.397099 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "009443acf2414655bb736bc7f7275afe" candidate_uuid: "20b9016f98a5411bb034d5e44b4dd52f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92c821b3841414c965660b54194112f"
I20260618 23:32:05.397204 8206 raft_consensus.cc:3037] T 009443acf2414655bb736bc7f7275afe P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:05.397202 8346 raft_consensus.cc:3037] T 009443acf2414655bb736bc7f7275afe P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:05.397925 8346 raft_consensus.cc:2454] T 009443acf2414655bb736bc7f7275afe P bd221a0a77b140149fa64959904c1af4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b9016f98a5411bb034d5e44b4dd52f in term 1.
I20260618 23:32:05.398109 8206 raft_consensus.cc:2454] T 009443acf2414655bb736bc7f7275afe P e92c821b3841414c965660b54194112f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20b9016f98a5411bb034d5e44b4dd52f in term 1.
I20260618 23:32:05.398113 7998 leader_election.cc:302] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20b9016f98a5411bb034d5e44b4dd52f, bd221a0a77b140149fa64959904c1af4; no voters:
I20260618 23:32:05.398342 8426 raft_consensus.cc:2781] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:32:05.398415 8426 raft_consensus.cc:686] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f [term 1 LEADER]: Becoming Leader. State: Replica: 20b9016f98a5411bb034d5e44b4dd52f, State: Running, Role: LEADER
I20260618 23:32:05.398494 8426 consensus_queue.cc:237] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.399237 7907 catalog_manager.cc:5495] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f reported cstate change: term changed from 0 to 1, leader changed from <none> to 20b9016f98a5411bb034d5e44b4dd52f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "20b9016f98a5411bb034d5e44b4dd52f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } health_report { overall_health: UNKNOWN } } }
I20260618 23:32:05.415158 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.412197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49338 (local address 127.0.98.1:36853)
0618 23:32:05.412260 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:05.412262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:05.412308 (+ 46us) server_negotiation.cc:407] Connection header received
0618 23:32:05.412414 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:05.412414 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:05.412427 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:05.412483 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:32:05.412668 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.413140 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.414318 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.414558 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.414618 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.414806 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:32:05.414835 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:32:05.414877 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:05.415024 (+ 147us) server_negotiation.cc:299] Negotiation successful
0618 23:32:05.415051 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 23:32:05.416085 8346 raft_consensus.cc:1261] T 009443acf2414655bb736bc7f7275afe P bd221a0a77b140149fa64959904c1af4 [term 1 FOLLOWER]: Refusing update from remote peer 20b9016f98a5411bb034d5e44b4dd52f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:32:05.416164 8206 raft_consensus.cc:1261] T 009443acf2414655bb736bc7f7275afe P e92c821b3841414c965660b54194112f [term 1 FOLLOWER]: Refusing update from remote peer 20b9016f98a5411bb034d5e44b4dd52f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:32:05.416438 8426 consensus_queue.cc:1035] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:32:05.416618 8431 consensus_queue.cc:1035] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:32:05.419278 7907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:32:05.419390 7907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:32:05.420235 7907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f: Sending DeleteTablet for 3 replicas of tablet 009443acf2414655bb736bc7f7275afe
I20260618 23:32:05.420540 8046 tablet_service.cc:1514] Processing DeleteTablet for tablet 009443acf2414655bb736bc7f7275afe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:32:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49322
I20260618 23:32:05.420667 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet 009443acf2414655bb736bc7f7275afe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:32:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34690
I20260618 23:32:05.420723 8444 tablet_replica.cc:330] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f: stopping tablet replica
I20260618 23:32:05.420758 8446 tablet_replica.cc:330] T 009443acf2414655bb736bc7f7275afe P e92c821b3841414c965660b54194112f: stopping tablet replica
I20260618 23:32:05.420810 8446 raft_consensus.cc:2229] T 009443acf2414655bb736bc7f7275afe P e92c821b3841414c965660b54194112f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:32:05.420806 8444 raft_consensus.cc:2229] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:32:05.420867 8446 raft_consensus.cc:2258] T 009443acf2414655bb736bc7f7275afe P e92c821b3841414c965660b54194112f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:32:05.420893 8444 raft_consensus.cc:2258] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:32:05.420997 8326 tablet_service.cc:1514] Processing DeleteTablet for tablet 009443acf2414655bb736bc7f7275afe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:32:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38958
I20260618 23:32:05.421162 8446 ts_tablet_manager.cc:1905] T 009443acf2414655bb736bc7f7275afe P e92c821b3841414c965660b54194112f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:32:05.421175 8445 tablet_replica.cc:330] T 009443acf2414655bb736bc7f7275afe P bd221a0a77b140149fa64959904c1af4: stopping tablet replica
I20260618 23:32:05.421255 8445 raft_consensus.cc:2229] T 009443acf2414655bb736bc7f7275afe P bd221a0a77b140149fa64959904c1af4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:32:05.421329 8445 raft_consensus.cc:2258] T 009443acf2414655bb736bc7f7275afe P bd221a0a77b140149fa64959904c1af4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:32:05.421653 8444 ts_tablet_manager.cc:1905] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:32:05.422798 8446 ts_tablet_manager.cc:1918] T 009443acf2414655bb736bc7f7275afe P e92c821b3841414c965660b54194112f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:32:05.422879 8446 log.cc:1196] T 009443acf2414655bb736bc7f7275afe P e92c821b3841414c965660b54194112f: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/009443acf2414655bb736bc7f7275afe
I20260618 23:32:05.423085 8446 ts_tablet_manager.cc:1939] T 009443acf2414655bb736bc7f7275afe P e92c821b3841414c965660b54194112f: Deleting consensus metadata
I20260618 23:32:05.423413 7894 catalog_manager.cc:4841] TS e92c821b3841414c965660b54194112f (127.0.98.2:40739): tablet 009443acf2414655bb736bc7f7275afe (table test-table [id=364cc2e74c5149d5ab253b5c286d839d]) successfully deleted
I20260618 23:32:05.423519 8444 ts_tablet_manager.cc:1918] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:32:05.423641 8444 log.cc:1196] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-0/wal/wals/009443acf2414655bb736bc7f7275afe
I20260618 23:32:05.423627 7907 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:32:05.423789 7907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:32:05.423892 8444 ts_tablet_manager.cc:1939] T 009443acf2414655bb736bc7f7275afe P 20b9016f98a5411bb034d5e44b4dd52f: Deleting consensus metadata
I20260618 23:32:05.424242 7893 catalog_manager.cc:4841] TS 20b9016f98a5411bb034d5e44b4dd52f (127.0.98.1:36853): tablet 009443acf2414655bb736bc7f7275afe (table test-table [id=364cc2e74c5149d5ab253b5c286d839d]) successfully deleted
I20260618 23:32:05.424563 8445 ts_tablet_manager.cc:1905] T 009443acf2414655bb736bc7f7275afe P bd221a0a77b140149fa64959904c1af4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:32:05.425066 8046 tablet_service.cc:1467] Processing CreateTablet for tablet 5515c788d4604780b1b3617c572f9fd5 (DEFAULT_TABLE table=test-table [id=f98daf14735b46e3ae340e29333a3d09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:32:05.425199 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5515c788d4604780b1b3617c572f9fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:05.425184 8326 tablet_service.cc:1467] Processing CreateTablet for tablet 5515c788d4604780b1b3617c572f9fd5 (DEFAULT_TABLE table=test-table [id=f98daf14735b46e3ae340e29333a3d09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:32:05.425288 8326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5515c788d4604780b1b3617c572f9fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:05.426345 8445 ts_tablet_manager.cc:1918] T 009443acf2414655bb736bc7f7275afe P bd221a0a77b140149fa64959904c1af4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:32:05.426399 8445 log.cc:1196] T 009443acf2414655bb736bc7f7275afe P bd221a0a77b140149fa64959904c1af4: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/009443acf2414655bb736bc7f7275afe
I20260618 23:32:05.426636 8445 ts_tablet_manager.cc:1939] T 009443acf2414655bb736bc7f7275afe P bd221a0a77b140149fa64959904c1af4: Deleting consensus metadata
I20260618 23:32:05.426836 8421 tablet_bootstrap.cc:492] T 5515c788d4604780b1b3617c572f9fd5 P 20b9016f98a5411bb034d5e44b4dd52f: Bootstrap starting.
I20260618 23:32:05.426918 8422 tablet_bootstrap.cc:492] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4: Bootstrap starting.
I20260618 23:32:05.427201 8421 tablet_bootstrap.cc:654] T 5515c788d4604780b1b3617c572f9fd5 P 20b9016f98a5411bb034d5e44b4dd52f: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:05.427593 8422 tablet_bootstrap.cc:654] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:05.427714 7894 catalog_manager.cc:4841] TS bd221a0a77b140149fa64959904c1af4 (127.0.98.3:43095): tablet 009443acf2414655bb736bc7f7275afe (table test-table [id=364cc2e74c5149d5ab253b5c286d839d]) successfully deleted
I20260618 23:32:05.427938 8186 tablet_service.cc:1467] Processing CreateTablet for tablet 5515c788d4604780b1b3617c572f9fd5 (DEFAULT_TABLE table=test-table [id=f98daf14735b46e3ae340e29333a3d09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:32:05.428088 8186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5515c788d4604780b1b3617c572f9fd5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:05.428484 8421 tablet_bootstrap.cc:492] T 5515c788d4604780b1b3617c572f9fd5 P 20b9016f98a5411bb034d5e44b4dd52f: No bootstrap required, opened a new log
I20260618 23:32:05.428552 8421 ts_tablet_manager.cc:1397] T 5515c788d4604780b1b3617c572f9fd5 P 20b9016f98a5411bb034d5e44b4dd52f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:32:05.428669 8421 raft_consensus.cc:348] T 5515c788d4604780b1b3617c572f9fd5 P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.428754 8421 raft_consensus.cc:374] T 5515c788d4604780b1b3617c572f9fd5 P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:05.428782 8421 raft_consensus.cc:729] T 5515c788d4604780b1b3617c572f9fd5 P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b9016f98a5411bb034d5e44b4dd52f, State: Initialized, Role: FOLLOWER
I20260618 23:32:05.428829 8421 consensus_queue.cc:260] T 5515c788d4604780b1b3617c572f9fd5 P 20b9016f98a5411bb034d5e44b4dd52f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.428929 8421 ts_tablet_manager.cc:1428] T 5515c788d4604780b1b3617c572f9fd5 P 20b9016f98a5411bb034d5e44b4dd52f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260618 23:32:05.429450 8422 tablet_bootstrap.cc:492] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4: No bootstrap required, opened a new log
I20260618 23:32:05.429499 8422 ts_tablet_manager.cc:1397] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260618 23:32:05.429563 8420 tablet_bootstrap.cc:492] T 5515c788d4604780b1b3617c572f9fd5 P e92c821b3841414c965660b54194112f: Bootstrap starting.
I20260618 23:32:05.429646 8422 raft_consensus.cc:348] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.429715 8422 raft_consensus.cc:374] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:05.429735 8422 raft_consensus.cc:729] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd221a0a77b140149fa64959904c1af4, State: Initialized, Role: FOLLOWER
I20260618 23:32:05.429790 8422 consensus_queue.cc:260] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.429886 8422 ts_tablet_manager.cc:1428] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:05.430022 8420 tablet_bootstrap.cc:654] T 5515c788d4604780b1b3617c572f9fd5 P e92c821b3841414c965660b54194112f: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:05.430745 8420 tablet_bootstrap.cc:492] T 5515c788d4604780b1b3617c572f9fd5 P e92c821b3841414c965660b54194112f: No bootstrap required, opened a new log
I20260618 23:32:05.430789 8420 ts_tablet_manager.cc:1397] T 5515c788d4604780b1b3617c572f9fd5 P e92c821b3841414c965660b54194112f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:05.430953 8420 raft_consensus.cc:348] T 5515c788d4604780b1b3617c572f9fd5 P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.431105 8420 raft_consensus.cc:374] T 5515c788d4604780b1b3617c572f9fd5 P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:05.431149 8420 raft_consensus.cc:729] T 5515c788d4604780b1b3617c572f9fd5 P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92c821b3841414c965660b54194112f, State: Initialized, Role: FOLLOWER
I20260618 23:32:05.431228 8420 consensus_queue.cc:260] T 5515c788d4604780b1b3617c572f9fd5 P e92c821b3841414c965660b54194112f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.431331 8420 ts_tablet_manager.cc:1428] T 5515c788d4604780b1b3617c572f9fd5 P e92c821b3841414c965660b54194112f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260618 23:32:05.498168 8427 raft_consensus.cc:482] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:32:05.498286 8427 raft_consensus.cc:504] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.498745 8427 leader_election.cc:288] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20b9016f98a5411bb034d5e44b4dd52f (127.0.98.1:36853), e92c821b3841414c965660b54194112f (127.0.98.2:40739)
I20260618 23:32:05.503238 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.499203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37871 (local address 127.0.98.2:40739)
0618 23:32:05.499270 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:05.499272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:05.499283 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:32:05.499383 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:05.499384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:05.499394 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:05.499464 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:05.499782 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.500556 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.502346 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.502784 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.502848 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.502857 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:05.503089 (+ 232us) server_negotiation.cc:299] Negotiation successful
0618 23:32:05.503130 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 23:32:05.503366 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.499037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35957 (local address 127.0.98.1:36853)
0618 23:32:05.499086 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:05.499089 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:05.499215 (+ 126us) server_negotiation.cc:407] Connection header received
0618 23:32:05.499388 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:05.499390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:05.499399 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:05.499464 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:05.499783 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.500547 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.502397 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.502911 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.502963 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.502977 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:05.503210 (+ 233us) server_negotiation.cc:299] Negotiation successful
0618 23:32:05.503282 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260618 23:32:05.503232 8501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.499021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40739 (local address 127.0.98.3:37871)
0618 23:32:05.499187 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:05.499200 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:05.499287 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:05.499494 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:05.499495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:05.499498 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:05.499690 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.499693 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.500681 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.500682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.502249 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.502252 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.502895 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.502896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.502976 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.502983 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 23:32:05.503052 (+ 69us) client_negotiation.cc:240] Negotiation successful
0618 23:32:05.503102 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":113,"thread_start_us":107,"threads_started":1}
I20260618 23:32:05.503338 8500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.498976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36853 (local address 127.0.98.3:35957)
0618 23:32:05.499138 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:05.499150 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:05.499248 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:05.499495 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:05.499496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:05.499498 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:05.499691 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.499696 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.500681 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.500682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.502249 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.502252 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.503037 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.503038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.503100 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.503103 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:32:05.503154 (+ 51us) client_negotiation.cc:240] Negotiation successful
0618 23:32:05.503187 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":94,"thread_start_us":80,"threads_started":1}
I20260618 23:32:05.503925 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5515c788d4604780b1b3617c572f9fd5" candidate_uuid: "bd221a0a77b140149fa64959904c1af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92c821b3841414c965660b54194112f" is_pre_election: true
I20260618 23:32:05.504033 8206 raft_consensus.cc:2454] T 5515c788d4604780b1b3617c572f9fd5 P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd221a0a77b140149fa64959904c1af4 in term 0.
I20260618 23:32:05.504227 8278 leader_election.cc:302] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd221a0a77b140149fa64959904c1af4, e92c821b3841414c965660b54194112f; no voters:
I20260618 23:32:05.504261 8066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5515c788d4604780b1b3617c572f9fd5" candidate_uuid: "bd221a0a77b140149fa64959904c1af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b9016f98a5411bb034d5e44b4dd52f" is_pre_election: true
I20260618 23:32:05.504336 8066 raft_consensus.cc:2454] T 5515c788d4604780b1b3617c572f9fd5 P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd221a0a77b140149fa64959904c1af4 in term 0.
I20260618 23:32:05.504495 8427 raft_consensus.cc:2781] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:32:05.504563 8427 raft_consensus.cc:482] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:32:05.504596 8427 raft_consensus.cc:3037] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:05.505420 8427 raft_consensus.cc:504] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.505600 8427 leader_election.cc:288] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4 [CANDIDATE]: Term 1 election: Requested vote from peers 20b9016f98a5411bb034d5e44b4dd52f (127.0.98.1:36853), e92c821b3841414c965660b54194112f (127.0.98.2:40739)
I20260618 23:32:05.505770 8206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5515c788d4604780b1b3617c572f9fd5" candidate_uuid: "bd221a0a77b140149fa64959904c1af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e92c821b3841414c965660b54194112f"
I20260618 23:32:05.505856 8206 raft_consensus.cc:3037] T 5515c788d4604780b1b3617c572f9fd5 P e92c821b3841414c965660b54194112f [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:05.505826 8066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5515c788d4604780b1b3617c572f9fd5" candidate_uuid: "bd221a0a77b140149fa64959904c1af4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20b9016f98a5411bb034d5e44b4dd52f"
I20260618 23:32:05.505924 8066 raft_consensus.cc:3037] T 5515c788d4604780b1b3617c572f9fd5 P 20b9016f98a5411bb034d5e44b4dd52f [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:05.506608 8206 raft_consensus.cc:2454] T 5515c788d4604780b1b3617c572f9fd5 P e92c821b3841414c965660b54194112f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd221a0a77b140149fa64959904c1af4 in term 1.
I20260618 23:32:05.506623 8066 raft_consensus.cc:2454] T 5515c788d4604780b1b3617c572f9fd5 P 20b9016f98a5411bb034d5e44b4dd52f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd221a0a77b140149fa64959904c1af4 in term 1.
I20260618 23:32:05.506798 8278 leader_election.cc:302] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd221a0a77b140149fa64959904c1af4, e92c821b3841414c965660b54194112f; no voters:
I20260618 23:32:05.506932 8427 raft_consensus.cc:2781] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:32:05.507087 8427 raft_consensus.cc:686] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4 [term 1 LEADER]: Becoming Leader. State: Replica: bd221a0a77b140149fa64959904c1af4, State: Running, Role: LEADER
I20260618 23:32:05.507192 8427 consensus_queue.cc:237] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } }
I20260618 23:32:05.508051 7907 catalog_manager.cc:5495] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd221a0a77b140149fa64959904c1af4 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "bd221a0a77b140149fa64959904c1af4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd221a0a77b140149fa64959904c1af4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43095 } health_report { overall_health: HEALTHY } } }
I20260618 23:32:05.527997 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.525517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38972 (local address 127.0.98.3:43095)
0618 23:32:05.525571 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:05.525573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:05.525583 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:32:05.525631 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:05.525632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:05.525652 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:05.525732 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:32:05.525898 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.526426 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.527198 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.527415 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.527484 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.527674 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:32:05.527794 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:32:05.527897 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:05.527914 (+ 17us) server_negotiation.cc:299] Negotiation successful
0618 23:32:05.527931 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:32:05.534063 8419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.531362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54483 (local address 127.0.98.2:40739)
0618 23:32:05.531443 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:05.531445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:05.531515 (+ 70us) server_negotiation.cc:407] Connection header received
0618 23:32:05.531639 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:05.531640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:05.531652 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:05.531712 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:32:05.532008 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.532560 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.533324 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.533517 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.533592 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.533791 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:32:05.533822 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:32:05.533869 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:05.533965 (+ 96us) server_negotiation.cc:299] Negotiation successful
0618 23:32:05.533985 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 23:32:05.534090 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.531263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40739 (local address 127.0.98.3:54483)
0618 23:32:05.531433 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:05.531452 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:05.531512 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:05.531810 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:05.531811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:05.531813 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:32:05.531870 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.531871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.532652 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.532652 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.533239 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.533240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.533622 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.533624 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.533682 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.533696 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:32:05.533916 (+ 220us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:32:05.533917 (+ 1us) client_negotiation.cc:769] Sending connection context
0618 23:32:05.533964 (+ 47us) client_negotiation.cc:240] Negotiation successful
0618 23:32:05.533999 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":112,"threads_started":1}
I20260618 23:32:05.536616 8066 raft_consensus.cc:1261] T 5515c788d4604780b1b3617c572f9fd5 P 20b9016f98a5411bb034d5e44b4dd52f [term 1 FOLLOWER]: Refusing update from remote peer bd221a0a77b140149fa64959904c1af4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:32:05.536690 8206 raft_consensus.cc:1261] T 5515c788d4604780b1b3617c572f9fd5 P e92c821b3841414c965660b54194112f [term 1 FOLLOWER]: Refusing update from remote peer bd221a0a77b140149fa64959904c1af4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:32:05.536994 8502 consensus_queue.cc:1035] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e92c821b3841414c965660b54194112f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:32:05.537156 8427 consensus_queue.cc:1035] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20b9016f98a5411bb034d5e44b4dd52f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:32:05.545308 8418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.542781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38051 (local address 127.0.98.3:43095)
0618 23:32:05.542822 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:05.542824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:05.542839 (+ 15us) server_negotiation.cc:407] Connection header received
0618 23:32:05.542912 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:05.542913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:05.542925 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:05.542988 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:32:05.543145 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.543762 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.544542 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.544868 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.544934 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.545060 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:32:05.545095 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:32:05.545148 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:05.545206 (+ 58us) server_negotiation.cc:299] Negotiation successful
0618 23:32:05.545231 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260618 23:32:05.545334 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.542735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43095 (local address 127.0.98.2:38051)
0618 23:32:05.542798 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:05.542810 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:05.542851 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:05.542997 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:05.542998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:05.542999 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:32:05.543064 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.543066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.543844 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.543845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.544424 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:05.544426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:05.544933 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:05.544934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:05.544986 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:05.544993 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:32:05.545150 (+ 157us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:32:05.545152 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:32:05.545211 (+ 59us) client_negotiation.cc:240] Negotiation successful
0618 23:32:05.545259 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260618 23:32:05.557018 7907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:32:05.557161 7907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:32:05.557868 7907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P efdfd46124e04c92bb3c500fe404e02f: Sending DeleteTablet for 3 replicas of tablet 5515c788d4604780b1b3617c572f9fd5
I20260618 23:32:05.558162 8186 tablet_service.cc:1514] Processing DeleteTablet for tablet 5515c788d4604780b1b3617c572f9fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:32:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34690
I20260618 23:32:05.558168 8046 tablet_service.cc:1514] Processing DeleteTablet for tablet 5515c788d4604780b1b3617c572f9fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:32:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49322
I20260618 23:32:05.558281 8446 tablet_replica.cc:330] T 5515c788d4604780b1b3617c572f9fd5 P e92c821b3841414c965660b54194112f: stopping tablet replica
I20260618 23:32:05.558301 8444 tablet_replica.cc:330] T 5515c788d4604780b1b3617c572f9fd5 P 20b9016f98a5411bb034d5e44b4dd52f: stopping tablet replica
I20260618 23:32:05.558331 8446 raft_consensus.cc:2229] T 5515c788d4604780b1b3617c572f9fd5 P e92c821b3841414c965660b54194112f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:32:05.558359 8444 raft_consensus.cc:2229] T 5515c788d4604780b1b3617c572f9fd5 P 20b9016f98a5411bb034d5e44b4dd52f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:32:05.558369 8446 raft_consensus.cc:2258] T 5515c788d4604780b1b3617c572f9fd5 P e92c821b3841414c965660b54194112f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:32:05.558403 8444 raft_consensus.cc:2258] T 5515c788d4604780b1b3617c572f9fd5 P 20b9016f98a5411bb034d5e44b4dd52f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:32:05.558586 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 7981
I20260618 23:32:05.561451 8446 ts_tablet_manager.cc:1905] T 5515c788d4604780b1b3617c572f9fd5 P e92c821b3841414c965660b54194112f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:32:05.563041 8446 ts_tablet_manager.cc:1918] T 5515c788d4604780b1b3617c572f9fd5 P e92c821b3841414c965660b54194112f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:32:05.563117 8446 log.cc:1196] T 5515c788d4604780b1b3617c572f9fd5 P e92c821b3841414c965660b54194112f: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/5515c788d4604780b1b3617c572f9fd5
I20260618 23:32:05.563366 8446 ts_tablet_manager.cc:1939] T 5515c788d4604780b1b3617c572f9fd5 P e92c821b3841414c965660b54194112f: Deleting consensus metadata
I20260618 23:32:05.563846 7894 catalog_manager.cc:4841] TS e92c821b3841414c965660b54194112f (127.0.98.2:40739): tablet 5515c788d4604780b1b3617c572f9fd5 (table test-table [id=f98daf14735b46e3ae340e29333a3d09]) successfully deleted
I20260618 23:32:05.566826 8326 tablet_service.cc:1514] Processing DeleteTablet for tablet 5515c788d4604780b1b3617c572f9fd5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:32:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38958
I20260618 23:32:05.566954 8445 tablet_replica.cc:330] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4: stopping tablet replica
I20260618 23:32:05.567018 8445 raft_consensus.cc:2229] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:32:05.567148 8445 raft_consensus.cc:2258] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260618 23:32:05.567319 7893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36853 (error 108)
I20260618 23:32:05.567610 8445 ts_tablet_manager.cc:1905] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:32:05.567914 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:05.567721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36853 (local address 127.0.0.1:49350)
0618 23:32:05.567825 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:05.567869 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":71,"threads_started":1}
I20260618 23:32:05.568054 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 8121
W20260618 23:32:05.570343 7893 catalog_manager.cc:4584] TS 20b9016f98a5411bb034d5e44b4dd52f (127.0.98.1:36853): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5515c788d4604780b1b3617c572f9fd5: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36853: connect: Connection refused (error 111)
I20260618 23:32:05.570698 8445 ts_tablet_manager.cc:1918] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:32:05.570779 8445 log.cc:1196] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/5515c788d4604780b1b3617c572f9fd5
I20260618 23:32:05.571013 8445 ts_tablet_manager.cc:1939] T 5515c788d4604780b1b3617c572f9fd5 P bd221a0a77b140149fa64959904c1af4: Deleting consensus metadata
I20260618 23:32:05.571414 7894 catalog_manager.cc:4841] TS bd221a0a77b140149fa64959904c1af4 (127.0.98.3:43095): tablet 5515c788d4604780b1b3617c572f9fd5 (table test-table [id=f98daf14735b46e3ae340e29333a3d09]) successfully deleted
I20260618 23:32:05.576375 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 8261
I20260618 23:32:05.582158 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 7877
2026-06-18T23:32:05Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2737 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:32:05 dist-test-slave-6224 krb5kdc[8516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:32:05 dist-test-slave-6224 krb5kdc[8516](info): set up 2 sockets
Jun 18 23:32:05 dist-test-slave-6224 krb5kdc[8516](info): commencing operation
krb5kdc: starting...
W20260618 23:32:06.624266 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:32:06 dist-test-slave-6224 krb5kdc[8516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:32:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:32:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:32:06.754841 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41821
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41821
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:32:06.842003 8529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:06.842274 8529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:06.842314 8529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:06.843573 8529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:32:06.843657 8529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:06.843683 8529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:06.843704 8529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:32:06.843741 8529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:32:06.845368 8529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8529
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:06.845665 8529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:06.845921 8529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:06.851434 8537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:06.851450 8535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:06.851732 8529 server_base.cc:1034] running on GCE node
W20260618 23:32:06.851451 8534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:06.852079 8529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:06.852401 8529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:06.853582 8529 hybrid_clock.cc:648] HybridClock initialized: now 1781825526853575 us; error 55 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-18T23:32:06Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1279 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:32:06 dist-test-slave-6224 krb5kdc[8542](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:32:06 dist-test-slave-6224 krb5kdc[8542](info): set up 2 sockets
Jun 18 23:32:06 dist-test-slave-6224 krb5kdc[8542](info): commencing operation
krb5kdc: starting...
W20260618 23:32:07.924252 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:32:08 dist-test-slave-6224 krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:32:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:32:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:32:08.050971 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42663
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:32:08.141217 8555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:08.141433 8555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:08.141455 8555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:08.142827 8555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:32:08.142892 8555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:08.142925 8555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:08.143018 8555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:32:08.143052 8555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:32:08.145553 8555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:08.145835 8555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:08.146153 8555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:08.151863 8561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:08.151902 8560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:08.151865 8563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:08.152133 8555 server_base.cc:1034] running on GCE node
I20260618 23:32:08.152463 8555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:08.152765 8555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:08.153971 8555 hybrid_clock.cc:648] HybridClock initialized: now 1781825528153961 us; error 45 us; skew 500 ppm
Jun 18 23:32:08 dist-test-slave-6224 krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:32:08.156630 8555 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:32:08.157176 8555 webserver.cc:460] Webserver started at http://127.0.98.62:42063/ using document root <none> and password file <none>
I20260618 23:32:08.157424 8555 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:08.157491 8555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:08.157635 8555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:32:08.158702 8555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "20d0084801454ba699f69e8dd42938a3"
format_stamp: "Formatted at 2026-06-18 23:32:08 on dist-test-slave-6224"
server_key: "929ffc3a16ad634ac63227181fd252d3"
server_key_iv: "2b84f9d8e8b369f66283dd9039f3c978"
server_key_version: "encryptionkey@0"
I20260618 23:32:08.159076 8555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "20d0084801454ba699f69e8dd42938a3"
format_stamp: "Formatted at 2026-06-18 23:32:08 on dist-test-slave-6224"
server_key: "929ffc3a16ad634ac63227181fd252d3"
server_key_iv: "2b84f9d8e8b369f66283dd9039f3c978"
server_key_version: "encryptionkey@0"
I20260618 23:32:08.160575 8555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:32:08.161378 8570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:08.161607 8555 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:08.161716 8555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "20d0084801454ba699f69e8dd42938a3"
format_stamp: "Formatted at 2026-06-18 23:32:08 on dist-test-slave-6224"
server_key: "929ffc3a16ad634ac63227181fd252d3"
server_key_iv: "2b84f9d8e8b369f66283dd9039f3c978"
server_key_version: "encryptionkey@0"
I20260618 23:32:08.161799 8555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:08.190837 8555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:08.192876 8555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:08.193075 8555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:08.197714 8555 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42663
I20260618 23:32:08.197820 8622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42663 every 8 connection(s)
I20260618 23:32:08.198319 8555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:32:08.199115 8623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:08.201731 8623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3: Bootstrap starting.
I20260618 23:32:08.202382 8623 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:08.202651 8623 log.cc:824] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3: Log is configured to *not* fsync() on all Append() calls
I20260618 23:32:08.203420 8623 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3: No bootstrap required, opened a new log
I20260618 23:32:08.205150 8623 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d0084801454ba699f69e8dd42938a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42663 } }
I20260618 23:32:08.205303 8623 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:08.205328 8623 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d0084801454ba699f69e8dd42938a3, State: Initialized, Role: FOLLOWER
I20260618 23:32:08.205446 8623 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d0084801454ba699f69e8dd42938a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42663 } }
I20260618 23:32:08.205502 8623 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:32:08.205525 8623 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:32:08.205554 8623 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:08.206461 8623 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d0084801454ba699f69e8dd42938a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42663 } }
I20260618 23:32:08.206550 8623 leader_election.cc:302] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20d0084801454ba699f69e8dd42938a3; no voters:
I20260618 23:32:08.206684 8623 leader_election.cc:288] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:32:08.206790 8626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:32:08.206991 8623 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:32:08.207024 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 8555
I20260618 23:32:08.207005 8626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [term 1 LEADER]: Becoming Leader. State: Replica: 20d0084801454ba699f69e8dd42938a3, State: Running, Role: LEADER
I20260618 23:32:08.207125 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:32:08.207157 8626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d0084801454ba699f69e8dd42938a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42663 } }
I20260618 23:32:08.207360 392 external_mini_cluster.cc:1428] Setting key b8b5d6103c874960ec180d3235f878f9
I20260618 23:32:08.207598 8628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20d0084801454ba699f69e8dd42938a3. Latest consensus state: current_term: 1 leader_uuid: "20d0084801454ba699f69e8dd42938a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d0084801454ba699f69e8dd42938a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42663 } } }
I20260618 23:32:08.207712 8628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [sys.catalog]: This master's current role is: LEADER
I20260618 23:32:08.207882 8627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20d0084801454ba699f69e8dd42938a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d0084801454ba699f69e8dd42938a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42663 } } }
I20260618 23:32:08.207958 8627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [sys.catalog]: This master's current role is: LEADER
I20260618 23:32:08.208069 8635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:32:08.208704 8635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:32:08.210568 8635 catalog_manager.cc:1347] Generated new cluster ID: ea1d303d3dd74b0b8d6b26eec10d9d05
I20260618 23:32:08.210620 8635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 18 23:32:08 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:32:08.220986 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:08.208099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51366 (local address 127.0.98.62:42663)
0618 23:32:08.208550 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:08.208562 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:08.208600 (+ 38us) server_negotiation.cc:407] Connection header received
0618 23:32:08.208749 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:08.208757 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:08.208827 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:08.209113 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:08.209974 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:08.211372 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:08.212578 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:08.213255 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:08.213391 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:08.216687 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:08.216692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:08.216692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:08.216705 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:08.218890 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:08.219393 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:08.219396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:08.219396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:08.219524 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:08.219781 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:08.219782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:08.219783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:08.220134 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:08.220301 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:08.220430 (+ 129us) server_negotiation.cc:299] Negotiation successful
0618 23:32:08.220705 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":106,"threads_started":1}
I20260618 23:32:08.230811 8635 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:32:08.231467 8635 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:32:08.238812 8635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3: Generated new TSK 0
I20260618 23:32:08.239073 8635 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260618 23:32:08.270534 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42663
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:32:08.360509 8649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:08.360798 8649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:08.360841 8649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:08.362047 8649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:08.362111 8649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:08.362166 8649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:32:08.363801 8649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:08.364092 8649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:08.364356 8649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:08.370123 8654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:08.370115 8657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:08.370116 8655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:08.370515 8649 server_base.cc:1034] running on GCE node
I20260618 23:32:08.370791 8649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:08.371096 8649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:08.372272 8649 hybrid_clock.cc:648] HybridClock initialized: now 1781825528372247 us; error 34 us; skew 500 ppm
Jun 18 23:32:08 dist-test-slave-6224 krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:32:08.375149 8649 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:32:08.375830 8649 webserver.cc:460] Webserver started at http://127.0.98.1:34637/ using document root <none> and password file <none>
I20260618 23:32:08.376108 8649 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:08.376181 8649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:08.376346 8649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:32:08.377553 8649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "7d5399d26099498cb94fc0cb26c62921"
format_stamp: "Formatted at 2026-06-18 23:32:08 on dist-test-slave-6224"
server_key: "da82f41701f0eddbd76664c5320af838"
server_key_iv: "6cff9f8fbff6c01790be86535ef12a6e"
server_key_version: "encryptionkey@0"
I20260618 23:32:08.377960 8649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7d5399d26099498cb94fc0cb26c62921"
format_stamp: "Formatted at 2026-06-18 23:32:08 on dist-test-slave-6224"
server_key: "da82f41701f0eddbd76664c5320af838"
server_key_iv: "6cff9f8fbff6c01790be86535ef12a6e"
server_key_version: "encryptionkey@0"
I20260618 23:32:08.379472 8649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:32:08.380379 8664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:08.380625 8649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:08.380738 8649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "7d5399d26099498cb94fc0cb26c62921"
format_stamp: "Formatted at 2026-06-18 23:32:08 on dist-test-slave-6224"
server_key: "da82f41701f0eddbd76664c5320af838"
server_key_iv: "6cff9f8fbff6c01790be86535ef12a6e"
server_key_version: "encryptionkey@0"
I20260618 23:32:08.380823 8649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:08.389288 8649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:08.391223 8649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:08.391402 8649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:08.391673 8649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:32:08.392016 8649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:32:08.392071 8649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:08.392112 8649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:32:08.392138 8649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:08.399501 8649 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37095
I20260618 23:32:08.399574 8777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37095 every 8 connection(s)
I20260618 23:32:08.399945 8649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:32:08.406663 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 8649
I20260618 23:32:08.406805 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:32:08.407019 392 external_mini_cluster.cc:1428] Setting key f0a8de3d2bdac7f1fd4c4eef1820d212
Jun 18 23:32:08 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:32:08.411170 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:08.400815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60419 (local address 127.0.98.62:42663)
0618 23:32:08.400884 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:08.400892 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:08.401144 (+ 252us) server_negotiation.cc:407] Connection header received
0618 23:32:08.402249 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:08.402251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:08.402270 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:08.402320 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:08.403031 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:08.403661 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:08.404740 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:08.404973 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:08.405066 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:08.407926 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:08.407928 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:08.407928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:08.407940 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:08.409532 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:08.410100 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:08.410101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:08.410102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:08.410162 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:08.410444 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:08.410445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:08.410446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:08.410578 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:08.410657 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:08.410973 (+ 316us) server_negotiation.cc:299] Negotiation successful
0618 23:32:08.411070 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 23:32:08.411412 8780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:08.400779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.1:60419)
0618 23:32:08.400992 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:08.401032 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:08.402108 (+ 1076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:08.402418 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:08.402423 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:08.402712 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:08.402931 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:08.402935 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:08.403762 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:08.403763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:08.404634 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:08.404637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:08.405075 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:08.405077 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:08.405220 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:08.405770 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:08.405781 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:08.407725 (+ 1944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:08.409654 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:08.409666 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:08.409670 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:08.410010 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:08.410245 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:08.410246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:08.410246 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:08.410370 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:08.410649 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:08.410656 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:08.410827 (+ 171us) client_negotiation.cc:769] Sending connection context
0618 23:32:08.410953 (+ 126us) client_negotiation.cc:240] Negotiation successful
0618 23:32:08.411146 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":111,"threads_started":1}
I20260618 23:32:08.411945 8778 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42663
I20260618 23:32:08.412086 8778 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:08.412350 8778 heartbeater.cc:511] Master 127.0.98.62:42663 requested a full tablet report, sending...
I20260618 23:32:08.413004 8587 ts_manager.cc:194] Registered new tserver with Master: 7d5399d26099498cb94fc0cb26c62921 (127.0.98.1:37095)
I20260618 23:32:08.413646 8587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60419
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260618 23:32:08.435231 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42663
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:32:08.525728 8783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:08.526012 8783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:08.526042 8783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:08.527415 8783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:08.527467 8783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:08.527498 8783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:32:08.529254 8783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:08.529573 8783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:08.529865 8783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:08.535642 8788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:08.535653 8791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:08.535722 8789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:08.535893 8783 server_base.cc:1034] running on GCE node
I20260618 23:32:08.536259 8783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:08.536502 8783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:08.537678 8783 hybrid_clock.cc:648] HybridClock initialized: now 1781825528537646 us; error 46 us; skew 500 ppm
Jun 18 23:32:08 dist-test-slave-6224 krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:32:08.540884 8783 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:32:08.541389 8783 webserver.cc:460] Webserver started at http://127.0.98.2:41917/ using document root <none> and password file <none>
I20260618 23:32:08.541631 8783 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:08.541692 8783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:08.541788 8783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:32:08.542869 8783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "e68babbbbee04648a74f01d3d224ac1c"
format_stamp: "Formatted at 2026-06-18 23:32:08 on dist-test-slave-6224"
server_key: "6c4c3f1e9bba136d2769bcffd398535d"
server_key_iv: "68a9e3ec47e83af70fb69f754b1e7fb1"
server_key_version: "encryptionkey@0"
I20260618 23:32:08.543190 8783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e68babbbbee04648a74f01d3d224ac1c"
format_stamp: "Formatted at 2026-06-18 23:32:08 on dist-test-slave-6224"
server_key: "6c4c3f1e9bba136d2769bcffd398535d"
server_key_iv: "68a9e3ec47e83af70fb69f754b1e7fb1"
server_key_version: "encryptionkey@0"
I20260618 23:32:08.544752 8783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:32:08.545651 8798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:08.545985 8783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:32:08.546073 8783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "e68babbbbee04648a74f01d3d224ac1c"
format_stamp: "Formatted at 2026-06-18 23:32:08 on dist-test-slave-6224"
server_key: "6c4c3f1e9bba136d2769bcffd398535d"
server_key_iv: "68a9e3ec47e83af70fb69f754b1e7fb1"
server_key_version: "encryptionkey@0"
I20260618 23:32:08.546205 8783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:08.555449 8783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:08.557770 8783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:08.557950 8783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:08.558197 8783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:32:08.558524 8783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:32:08.558580 8783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:08.558615 8783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:32:08.558645 8783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:08.565443 8783 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41979
I20260618 23:32:08.565521 8911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41979 every 8 connection(s)
I20260618 23:32:08.565897 8783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:32:08.570993 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 8783
I20260618 23:32:08.571110 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:32:08.571362 392 external_mini_cluster.cc:1428] Setting key 46661534b19039470d4396d5f9b27977
Jun 18 23:32:08 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:32:08.576845 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:08.566846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41553 (local address 127.0.98.62:42663)
0618 23:32:08.566943 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:08.566945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:08.567179 (+ 234us) server_negotiation.cc:407] Connection header received
0618 23:32:08.568310 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:08.568311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:08.568324 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:08.568378 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:08.569045 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:08.569570 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:08.570577 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:08.570782 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:08.570869 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:08.573630 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:08.573631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:08.573631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:08.573638 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:08.575330 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:08.575840 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:08.575841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:08.575841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:08.575898 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:08.576181 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:08.576182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:08.576182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:08.576300 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:08.576374 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:08.576659 (+ 285us) server_negotiation.cc:299] Negotiation successful
0618 23:32:08.576751 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260618 23:32:08.577085 8914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:08.566811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.2:41553)
0618 23:32:08.567027 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:08.567076 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:08.568190 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:08.568478 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:08.568482 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:08.568740 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:08.568943 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:08.568947 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:08.569668 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:08.569669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:08.570481 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:08.570483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:08.570879 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:08.570880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:08.571013 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:08.571676 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:08.571688 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:08.573506 (+ 1818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:08.575411 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:08.575418 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:08.575421 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:08.575747 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:08.575967 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:08.575968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:08.575968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:08.576106 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:08.576364 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:08.576370 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:08.576515 (+ 145us) client_negotiation.cc:769] Sending connection context
0618 23:32:08.576627 (+ 112us) client_negotiation.cc:240] Negotiation successful
0618 23:32:08.576804 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":101,"threads_started":1}
I20260618 23:32:08.577524 8912 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42663
I20260618 23:32:08.577627 8912 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:08.577832 8912 heartbeater.cc:511] Master 127.0.98.62:42663 requested a full tablet report, sending...
I20260618 23:32:08.578208 8587 ts_manager.cc:194] Registered new tserver with Master: e68babbbbee04648a74f01d3d224ac1c (127.0.98.2:41979)
I20260618 23:32:08.578608 8587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41553
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260618 23:32:08.600806 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42663
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:32:08.693470 8917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:08.693728 8917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:08.693758 8917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:08.695159 8917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:08.695211 8917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:08.695242 8917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:32:08.697008 8917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:08.697304 8917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:08.697595 8917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:08.703182 8925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:08.703200 8922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:08.703318 8923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:08.703580 8917 server_base.cc:1034] running on GCE node
I20260618 23:32:08.703830 8917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:08.704106 8917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:08.705255 8917 hybrid_clock.cc:648] HybridClock initialized: now 1781825528705229 us; error 44 us; skew 500 ppm
Jun 18 23:32:08 dist-test-slave-6224 krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:32:08.708364 8917 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:32:08.708987 8917 webserver.cc:460] Webserver started at http://127.0.98.3:44709/ using document root <none> and password file <none>
I20260618 23:32:08.709244 8917 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:08.709321 8917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:08.709462 8917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:32:08.710651 8917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "77db95fdad8340a6952c1922783beb41"
format_stamp: "Formatted at 2026-06-18 23:32:08 on dist-test-slave-6224"
server_key: "5f30e0a70f38946f2694ca8f38a589a7"
server_key_iv: "df005ee88c24844e034365dcdc1633cd"
server_key_version: "encryptionkey@0"
I20260618 23:32:08.711051 8917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "77db95fdad8340a6952c1922783beb41"
format_stamp: "Formatted at 2026-06-18 23:32:08 on dist-test-slave-6224"
server_key: "5f30e0a70f38946f2694ca8f38a589a7"
server_key_iv: "df005ee88c24844e034365dcdc1633cd"
server_key_version: "encryptionkey@0"
I20260618 23:32:08.712659 8917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:32:08.713526 8932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:08.713747 8917 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:08.713855 8917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "77db95fdad8340a6952c1922783beb41"
format_stamp: "Formatted at 2026-06-18 23:32:08 on dist-test-slave-6224"
server_key: "5f30e0a70f38946f2694ca8f38a589a7"
server_key_iv: "df005ee88c24844e034365dcdc1633cd"
server_key_version: "encryptionkey@0"
I20260618 23:32:08.713949 8917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:08.728248 8917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:08.730734 8917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:08.730924 8917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:08.731202 8917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:32:08.731670 8917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:32:08.731727 8917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:08.731770 8917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:32:08.731786 8917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:08.739723 8917 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40939
I20260618 23:32:08.739833 9045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40939 every 8 connection(s)
I20260618 23:32:08.740234 8917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:32:08.747138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 8917
I20260618 23:32:08.747246 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:32:08.747453 392 external_mini_cluster.cc:1428] Setting key 751aca8d2512be450cbee0a5128fa38d
Jun 18 23:32:08 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:32:08.751319 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:08.741233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37125 (local address 127.0.98.62:42663)
0618 23:32:08.741375 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:08.741378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:08.741571 (+ 193us) server_negotiation.cc:407] Connection header received
0618 23:32:08.742730 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:08.742731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:08.742745 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:08.742799 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:08.743506 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:08.744083 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:08.745173 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:08.745403 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:08.745492 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:08.748324 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:08.748326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:08.748326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:08.748335 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:08.749736 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:08.750383 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:08.750383 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:08.750384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:08.750441 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:08.750687 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:08.750688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:08.750688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:08.750804 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:08.750859 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:08.751162 (+ 303us) server_negotiation.cc:299] Negotiation successful
0618 23:32:08.751232 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260618 23:32:08.751543 9048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:08.741130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.3:37125)
0618 23:32:08.741395 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:08.741450 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:08.742593 (+ 1143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:08.742879 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:08.742883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:08.743186 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:08.743413 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:08.743417 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:08.744180 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:08.744181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:08.745068 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:08.745070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:08.745501 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:08.745502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:08.745653 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:08.746265 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:08.746273 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:08.748163 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:08.749843 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:08.749849 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:08.749851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:08.750151 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:08.750509 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:08.750510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:08.750510 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:08.750617 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:08.750864 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:08.750868 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:08.751036 (+ 168us) client_negotiation.cc:769] Sending connection context
0618 23:32:08.751132 (+ 96us) client_negotiation.cc:240] Negotiation successful
0618 23:32:08.751272 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":116,"threads_started":1}
I20260618 23:32:08.752174 9046 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42663
I20260618 23:32:08.752290 9046 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:08.752511 9046 heartbeater.cc:511] Master 127.0.98.62:42663 requested a full tablet report, sending...
I20260618 23:32:08.752955 8587 ts_manager.cc:194] Registered new tserver with Master: 77db95fdad8340a6952c1922783beb41 (127.0.98.3:40939)
I20260618 23:32:08.753350 8587 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37125
I20260618 23:32:08.759744 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260618 23:32:08.838095 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 8649
I20260618 23:32:08.843119 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 8783
I20260618 23:32:08.847855 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 8917
I20260618 23:32:08.852756 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 8555
I20260618 23:32:08.859330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42663
--webserver_interface=127.0.98.62
--webserver_port=42063
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:32:08.948695 9055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:08.948959 9055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:08.948985 9055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:08.950173 9055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:32:08.950214 9055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:08.950228 9055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:08.950241 9055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:32:08.950253 9055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:32:08.951934 9055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:08.952179 9055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:08.952397 9055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:08.958042 9061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:08.958024 9060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:08.958227 9055 server_base.cc:1034] running on GCE node
W20260618 23:32:08.958024 9063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:08.958617 9055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:08.958985 9055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:08.960175 9055 hybrid_clock.cc:648] HybridClock initialized: now 1781825528960166 us; error 37 us; skew 500 ppm
Jun 18 23:32:08 dist-test-slave-6224 krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825528, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:32:08.962965 9055 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:32:08.963472 9055 webserver.cc:460] Webserver started at http://127.0.98.62:42063/ using document root <none> and password file <none>
I20260618 23:32:08.963737 9055 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:08.963805 9055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:08.965389 9055 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:08.966156 9070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:08.966403 9055 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:08.966524 9055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "20d0084801454ba699f69e8dd42938a3"
format_stamp: "Formatted at 2026-06-18 23:32:08 on dist-test-slave-6224"
server_key: "929ffc3a16ad634ac63227181fd252d3"
server_key_iv: "2b84f9d8e8b369f66283dd9039f3c978"
server_key_version: "encryptionkey@0"
I20260618 23:32:08.966919 9055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:08.977429 9055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:08.979774 9055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:08.979941 9055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:08.984187 9055 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42663
I20260618 23:32:08.984606 9055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:32:08.984553 9122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42663 every 8 connection(s)
I20260618 23:32:08.985049 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 9055
I20260618 23:32:08.985361 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37095
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42663
--webserver_port=34637
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:32:08.986302 9123 sys_catalog.cc:263] Verifying existing consensus state
I20260618 23:32:08.987344 9123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3: Bootstrap starting.
I20260618 23:32:08.997521 9123 log.cc:824] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3: Log is configured to *not* fsync() on all Append() calls
I20260618 23:32:08.999799 9123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:32:09.000123 9123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3: Bootstrap complete.
I20260618 23:32:09.002526 9123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d0084801454ba699f69e8dd42938a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42663 } }
I20260618 23:32:09.002767 9123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20d0084801454ba699f69e8dd42938a3, State: Initialized, Role: FOLLOWER
I20260618 23:32:09.002965 9123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d0084801454ba699f69e8dd42938a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42663 } }
I20260618 23:32:09.003048 9123 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:32:09.003095 9123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:32:09.003145 9123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [term 1 FOLLOWER]: Advancing to term 2
I20260618 23:32:09.004343 9123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d0084801454ba699f69e8dd42938a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42663 } }
I20260618 23:32:09.004442 9123 leader_election.cc:302] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20d0084801454ba699f69e8dd42938a3; no voters:
I20260618 23:32:09.004694 9123 leader_election.cc:288] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260618 23:32:09.004733 9127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260618 23:32:09.004853 9127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [term 2 LEADER]: Becoming Leader. State: Replica: 20d0084801454ba699f69e8dd42938a3, State: Running, Role: LEADER
I20260618 23:32:09.005016 9123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:32:09.004985 9127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d0084801454ba699f69e8dd42938a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42663 } }
I20260618 23:32:09.005642 9129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20d0084801454ba699f69e8dd42938a3. Latest consensus state: current_term: 2 leader_uuid: "20d0084801454ba699f69e8dd42938a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d0084801454ba699f69e8dd42938a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42663 } } }
I20260618 23:32:09.005760 9129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [sys.catalog]: This master's current role is: LEADER
I20260618 23:32:09.005677 9128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "20d0084801454ba699f69e8dd42938a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20d0084801454ba699f69e8dd42938a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42663 } } }
I20260618 23:32:09.005987 9136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:32:09.006006 9128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3 [sys.catalog]: This master's current role is: LEADER
I20260618 23:32:09.006608 9136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:32:09.006757 9136 catalog_manager.cc:1259] Loaded cluster ID: ea1d303d3dd74b0b8d6b26eec10d9d05
I20260618 23:32:09.006795 9136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:32:09.008014 9136 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:32:09.008227 9136 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 20d0084801454ba699f69e8dd42938a3: Loaded TSK: 0
I20260618 23:32:09.008471 9136 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 23:32:09.072446 9125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:09.072698 9125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:09.072736 9125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:09.073956 9125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:09.074033 9125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:09.074079 9125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:32:09.075619 9125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34637
--tserver_master_addrs=127.0.98.62:42663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:09.075896 9125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:09.076126 9125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:09.081498 9149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:09.081494 9152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:09.081472 9150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:09.081988 9125 server_base.cc:1034] running on GCE node
I20260618 23:32:09.082206 9125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:09.082491 9125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:09.083643 9125 hybrid_clock.cc:648] HybridClock initialized: now 1781825529083626 us; error 28 us; skew 500 ppm
Jun 18 23:32:09 dist-test-slave-6224 krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825529, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:32:09.086262 9125 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260618 23:32:09.086777 9125 webserver.cc:460] Webserver started at http://127.0.98.1:34637/ using document root <none> and password file <none>
I20260618 23:32:09.087037 9125 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:09.087095 9125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:09.088655 9125 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:09.089363 9159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:09.089545 9125 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:09.089632 9125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "7d5399d26099498cb94fc0cb26c62921"
format_stamp: "Formatted at 2026-06-18 23:32:08 on dist-test-slave-6224"
server_key: "da82f41701f0eddbd76664c5320af838"
server_key_iv: "6cff9f8fbff6c01790be86535ef12a6e"
server_key_version: "encryptionkey@0"
I20260618 23:32:09.090001 9125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:09.102236 9125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:09.104557 9125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:09.104717 9125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:09.104987 9125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:32:09.105310 9125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:32:09.105365 9125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:09.105408 9125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:32:09.105434 9125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:09.112809 9125 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37095
I20260618 23:32:09.112838 9272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37095 every 8 connection(s)
I20260618 23:32:09.113204 9125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:32:09.121316 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 9125
I20260618 23:32:09.121676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41979
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42663
--webserver_port=41917
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 23:32:09 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:09 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:09.124138 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:09.113966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.1:59391)
0618 23:32:09.114192 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:09.114249 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:09.115051 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:09.115540 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:09.115570 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:09.116422 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:09.116731 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:09.116737 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.117942 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.117943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:09.119269 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:09.119272 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.119975 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.119976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:09.120144 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:09.120765 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:09.120775 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:09.123883 (+ 3108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":111,"threads_started":1}
W20260618 23:32:09.124166 9276 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:09.114156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59391 (local address 127.0.98.62:42663)
0618 23:32:09.114311 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:09.114321 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:09.114386 (+ 65us) server_negotiation.cc:407] Connection header received
0618 23:32:09.115264 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:09.115271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:09.115318 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:09.115442 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:09.116859 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.117826 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.119374 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.119876 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.119960 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:09.124013 (+ 4053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59391: BlockingRecv error: recv got EOF from 127.0.98.1:59391 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":75,"threads_started":1}
W20260618 23:32:09.124408 9273 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42663 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42663: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 18 23:32:09 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:09 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:09.133548 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:09.124650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.1:58811)
0618 23:32:09.124727 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:09.124739 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:09.124775 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:09.125082 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:09.125084 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:09.125224 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:09.125274 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:09.125276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.125908 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.125909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:09.126514 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:09.126516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.126885 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.126886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:09.126952 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:09.131682 (+ 4730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:09.131687 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:09.133460 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":61}
W20260618 23:32:09.133605 9276 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:09.124895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58811 (local address 127.0.98.62:42663)
0618 23:32:09.124953 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:09.124955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:09.124968 (+ 13us) server_negotiation.cc:407] Connection header received
0618 23:32:09.124988 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:09.124988 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:09.124997 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:09.125062 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:09.125349 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.125824 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.126599 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.126782 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.126887 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:09.133540 (+ 6653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58811: BlockingRecv error: recv got EOF from 127.0.98.1:58811 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 18 23:32:09 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:09 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:09.137993 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:09.133970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.1:49407)
0618 23:32:09.134014 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:09.134024 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:09.134063 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:09.134217 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:09.134218 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:09.134340 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:09.134369 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:09.134371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.134989 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.134990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:09.135476 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:09.135477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.135859 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.135860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:09.135928 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:09.136307 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:09.136312 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:09.137914 (+ 1602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260618 23:32:09.138026 9276 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:09.134007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49407 (local address 127.0.98.62:42663)
0618 23:32:09.134043 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:09.134044 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:09.134052 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:32:09.134132 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:09.134133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:09.134142 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:09.134213 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:09.134449 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.134910 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.135591 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.135774 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.135858 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:09.137972 (+ 2114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49407: BlockingRecv error: recv got EOF from 127.0.98.1:49407 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260618 23:32:09.138358 9273 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 23:32:09.214534 9277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:09.214748 9277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:09.214771 9277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:09.215879 9277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:09.215921 9277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:09.215947 9277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:32:09.217316 9277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41917
--tserver_master_addrs=127.0.98.62:42663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:09.217500 9277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:09.217674 9277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:09.222961 9282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:09.223037 9285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:09.223012 9283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:09.223272 9277 server_base.cc:1034] running on GCE node
I20260618 23:32:09.223471 9277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:09.223762 9277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:09.224910 9277 hybrid_clock.cc:648] HybridClock initialized: now 1781825529224896 us; error 42 us; skew 500 ppm
Jun 18 23:32:09 dist-test-slave-6224 krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825529, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:32:09.227828 9277 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260618 23:32:09.228361 9277 webserver.cc:460] Webserver started at http://127.0.98.2:41917/ using document root <none> and password file <none>
I20260618 23:32:09.228598 9277 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:09.228680 9277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:09.230159 9277 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:32:09.230938 9292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:09.231184 9277 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:32:09.231290 9277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "e68babbbbee04648a74f01d3d224ac1c"
format_stamp: "Formatted at 2026-06-18 23:32:08 on dist-test-slave-6224"
server_key: "6c4c3f1e9bba136d2769bcffd398535d"
server_key_iv: "68a9e3ec47e83af70fb69f754b1e7fb1"
server_key_version: "encryptionkey@0"
I20260618 23:32:09.231707 9277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:09.241047 9277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:09.243345 9277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:09.243515 9277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:09.243824 9277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:32:09.244155 9277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:32:09.244212 9277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:09.244251 9277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:32:09.244278 9277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:09.251340 9277 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41979
I20260618 23:32:09.251394 9405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41979 every 8 connection(s)
I20260618 23:32:09.251793 9277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
Jun 18 23:32:09 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:09.258721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 9277
I20260618 23:32:09.259217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:40939
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42663
--webserver_port=44709
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 23:32:09 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260618 23:32:09.260404 9276 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:09.252559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39413 (local address 127.0.98.62:42663)
0618 23:32:09.252610 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:09.252613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:09.252814 (+ 201us) server_negotiation.cc:407] Connection header received
0618 23:32:09.253354 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:09.253356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:09.253375 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:09.253464 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:09.254454 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.255003 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.256112 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.256326 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.256402 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:09.260345 (+ 3943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39413: BlockingRecv error: recv got EOF from 127.0.98.2:39413 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260618 23:32:09.260448 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:09.252504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.2:39413)
0618 23:32:09.252674 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:09.252717 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:09.253217 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:09.253543 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:09.253546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:09.254141 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:09.254336 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:09.254340 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.255114 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.255115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:09.255931 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:09.255933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.256457 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.256458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:09.256572 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:09.257176 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:09.257186 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:09.260241 (+ 3055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
W20260618 23:32:09.260684 9406 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42663 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42663: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 18 23:32:09 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:09 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:09.265511 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:09.260967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.2:39947)
0618 23:32:09.261023 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:09.261035 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:09.261096 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:09.261261 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:09.261261 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:09.261389 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:09.261433 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:09.261435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.262131 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.262132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:09.262633 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:09.262634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.263001 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.263002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:09.263072 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:09.263643 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:09.263647 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:09.265421 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":37}
W20260618 23:32:09.265569 9276 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:09.261056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39947 (local address 127.0.98.62:42663)
0618 23:32:09.261133 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:09.261136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:09.261145 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:32:09.261185 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:09.261186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:09.261195 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:09.261258 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:09.261520 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.262038 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.262718 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.262912 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.262994 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:09.265508 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39947: BlockingRecv error: recv got EOF from 127.0.98.2:39947 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 18 23:32:09 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:09 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:09.270087 9408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:09.265870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.2:49869)
0618 23:32:09.265918 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:09.265929 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:09.265973 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:09.266155 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:09.266156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:09.266279 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:09.266306 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:09.266307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.266918 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.266919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:09.267396 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:09.267398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.267835 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.267836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:09.267894 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:09.268367 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:09.268371 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:09.270001 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260618 23:32:09.270280 9276 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:09.265967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49869 (local address 127.0.98.62:42663)
0618 23:32:09.266044 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:09.266046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:09.266053 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:32:09.266073 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:09.266074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:09.266082 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:09.266151 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:09.266387 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.266842 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.267481 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.267753 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.267834 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:09.270076 (+ 2242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49869: BlockingRecv error: recv got EOF from 127.0.98.2:49869 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260618 23:32:09.270401 9406 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 23:32:09.350231 9409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:09.350495 9409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:09.350526 9409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:09.351962 9409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:09.352027 9409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:09.352063 9409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:32:09.353757 9409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:40939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44709
--tserver_master_addrs=127.0.98.62:42663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:09.354072 9409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:09.354387 9409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:09.360121 9414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:09.360160 9415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:09.360121 9417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:09.360479 9409 server_base.cc:1034] running on GCE node
I20260618 23:32:09.360719 9409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:09.361018 9409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:09.362210 9409 hybrid_clock.cc:648] HybridClock initialized: now 1781825529362205 us; error 51 us; skew 500 ppm
Jun 18 23:32:09 dist-test-slave-6224 krb5kdc[8542](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825529, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:32:09.365079 9409 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260618 23:32:09.365573 9409 webserver.cc:460] Webserver started at http://127.0.98.3:44709/ using document root <none> and password file <none>
I20260618 23:32:09.365833 9409 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:09.365895 9409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:09.367431 9409 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:32:09.368192 9424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:09.368474 9409 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260618 23:32:09.368583 9409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "77db95fdad8340a6952c1922783beb41"
format_stamp: "Formatted at 2026-06-18 23:32:08 on dist-test-slave-6224"
server_key: "5f30e0a70f38946f2694ca8f38a589a7"
server_key_iv: "df005ee88c24844e034365dcdc1633cd"
server_key_version: "encryptionkey@0"
I20260618 23:32:09.368988 9409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:09.403936 9409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:09.406445 9409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:09.406587 9409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:09.406858 9409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:32:09.407207 9409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:32:09.407248 9409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:09.407269 9409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:32:09.407282 9409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:09.414309 9409 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40939
I20260618 23:32:09.414438 9537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40939 every 8 connection(s)
I20260618 23:32:09.414762 9409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:32:09.415647 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 9409
Jun 18 23:32:09 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:09 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260618 23:32:09.423768 9276 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:09.415710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43917 (local address 127.0.98.62:42663)
0618 23:32:09.415774 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:09.415777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:09.416078 (+ 301us) server_negotiation.cc:407] Connection header received
0618 23:32:09.416863 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:09.416864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:09.416870 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:09.416916 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:09.418152 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.418689 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.420184 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.420440 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.420512 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:09.423692 (+ 3180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43917: BlockingRecv error: recv got EOF from 127.0.98.3:43917 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260618 23:32:09.423849 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:09.415703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.3:43917)
0618 23:32:09.415920 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:09.415977 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:09.416724 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:09.417007 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:09.417012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:09.417826 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:09.418051 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:09.418055 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.418803 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.418804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:09.420089 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:09.420092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.420532 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.420533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:09.420678 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:09.421148 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:09.421158 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:09.423620 (+ 2462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":103,"threads_started":1}
W20260618 23:32:09.424078 9538 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42663 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:42663: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260618 23:32:09.425344 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:09.416524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51368 (local address 127.0.98.62:42663)
0618 23:32:09.416645 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:09.416647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:09.416654 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:32:09.416680 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:09.416682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:09.416696 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:09.416752 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:09.417693 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.418346 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.419525 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.419776 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.419837 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:09.421565 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:09.421569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:09.421570 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:09.421580 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:09.423475 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:09.423947 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:09.423951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:09.423951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:09.424066 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:09.424528 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:09.424529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:09.424530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:09.424845 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:09.425005 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:09.425138 (+ 133us) server_negotiation.cc:299] Negotiation successful
0618 23:32:09.425230 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":77,"threads_started":1}
Jun 18 23:32:09 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:09 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:09.429621 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:09.424342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.3:47769)
0618 23:32:09.424418 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:09.424434 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:09.424484 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:09.424645 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:09.424646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:09.424823 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:09.424897 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:09.424900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.425814 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.425815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:09.426630 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:09.426633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.426986 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.426987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:09.427036 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:09.427482 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:09.427487 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:09.429569 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":48}
W20260618 23:32:09.429791 9276 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:09.424420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47769 (local address 127.0.98.62:42663)
0618 23:32:09.424504 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:09.424507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:09.424530 (+ 23us) server_negotiation.cc:407] Connection header received
0618 23:32:09.424559 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:09.424560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:09.424574 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:09.424616 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:09.425022 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.425736 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.426706 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.426888 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.426964 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:09.429684 (+ 2720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47769: BlockingRecv error: recv got EOF from 127.0.98.3:47769 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 18 23:32:09 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:09 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:09.435111 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:09.430430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.3:33371)
0618 23:32:09.430501 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:09.430516 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:09.430572 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:09.430694 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:09.430695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:09.430794 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:09.430820 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:09.430821 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.431506 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.431508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:09.432122 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:09.432124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.432501 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.432502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:09.432539 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:09.433027 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:09.433032 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:09.435060 (+ 2028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260618 23:32:09.435241 9276 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:09.430507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33371 (local address 127.0.98.62:42663)
0618 23:32:09.430575 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:09.430578 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:09.430589 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:32:09.430614 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:09.430614 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:09.430623 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:09.430680 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:09.430890 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.431399 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.432215 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:09.432420 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:09.432484 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:09.435152 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33371: BlockingRecv error: recv got EOF from 127.0.98.3:33371 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260618 23:32:09.435317 9538 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 18 23:32:10 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:10 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:10.144131 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:10.138818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.1:52703)
0618 23:32:10.138971 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:10.138990 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:10.139041 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:10.139353 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:10.139354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:10.139541 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:10.139630 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:10.139633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:10.140435 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:10.140436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:10.141009 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:10.141011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:10.141392 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:10.141393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:10.141459 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:10.141984 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:10.141989 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:10.144058 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
W20260618 23:32:10.144290 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:10.138947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52703 (local address 127.0.98.62:42663)
0618 23:32:10.139105 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:10.139108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:10.139116 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:32:10.139144 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:10.139144 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:10.139161 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:10.139271 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:10.139764 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:10.140331 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:10.141102 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:10.141296 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:10.141354 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:10.144182 (+ 2828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52703: BlockingRecv error: recv got EOF from 127.0.98.1:52703 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
Jun 18 23:32:10 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:10 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:10.276553 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:10.271011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.2:43861)
0618 23:32:10.271188 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:10.271201 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:10.271288 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:10.271475 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:10.271476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:10.271746 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:10.271835 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:10.271837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:10.272548 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:10.272549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:10.273247 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:10.273250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:10.273675 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:10.273676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:10.273737 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:10.274476 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:10.274482 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:10.276466 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
W20260618 23:32:10.276712 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:10.271097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43861 (local address 127.0.98.62:42663)
0618 23:32:10.271178 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:10.271180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:10.271275 (+ 95us) server_negotiation.cc:407] Connection header received
0618 23:32:10.271349 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:10.271349 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:10.271363 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:10.271430 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:10.271919 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:10.272427 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:10.273357 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:10.273565 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:10.273628 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:10.276617 (+ 2989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43861: BlockingRecv error: recv got EOF from 127.0.98.2:43861 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 18 23:32:10 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:10 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:10.441349 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:10.435938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.3:57841)
0618 23:32:10.436092 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:10.436110 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:10.436197 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:10.436433 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:10.436434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:10.436704 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:10.436795 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:10.436797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:10.437573 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:10.437574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:10.438189 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:10.438191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:10.438574 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:10.438575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:10.438628 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:10.439230 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:10.439241 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:10.441264 (+ 2023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":95,"threads_started":1}
W20260618 23:32:10.441522 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:10.436115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57841 (local address 127.0.98.62:42663)
0618 23:32:10.436233 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:10.436237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:10.436274 (+ 37us) server_negotiation.cc:407] Connection header received
0618 23:32:10.436305 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:10.436307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:10.436321 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:10.436411 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:10.436926 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:10.437455 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:10.438281 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:10.438462 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:10.438540 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:10.441407 (+ 2867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:57841: BlockingRecv error: recv got EOF from 127.0.98.3:57841 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 18 23:32:11 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:11 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:11.150014 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:11.144849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.1:45109)
0618 23:32:11.145066 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:11.145080 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:11.145168 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:11.145368 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:11.145369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:11.145537 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:11.145591 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:11.145592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:11.146316 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:11.146317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:11.146861 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:11.146863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:11.147301 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:11.147302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:11.147366 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:11.147956 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:11.147962 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:11.149910 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":124,"threads_started":1}
W20260618 23:32:11.150082 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:11.145005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45109 (local address 127.0.98.62:42663)
0618 23:32:11.145167 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:11.145169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:11.145181 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:32:11.145218 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:11.145219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:11.145231 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:11.145305 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:11.145682 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:11.146233 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:11.146945 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:11.147184 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:11.147258 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:11.150016 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45109: BlockingRecv error: recv got EOF from 127.0.98.1:45109 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":93,"threads_started":1}
Jun 18 23:32:11 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:11 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:11.282838 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:11.277160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.2:35995)
0618 23:32:11.277396 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:11.277415 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:11.277542 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:11.277746 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:11.277747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:11.277975 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:11.278034 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:11.278036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:11.278761 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:11.278762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:11.279413 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:11.279415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:11.279935 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:11.279936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:11.279996 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:11.280541 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:11.280546 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:11.282773 (+ 2227us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":103,"threads_started":1}
W20260618 23:32:11.282977 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:11.277295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35995 (local address 127.0.98.62:42663)
0618 23:32:11.277401 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:11.277405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:11.277513 (+ 108us) server_negotiation.cc:407] Connection header received
0618 23:32:11.277635 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:11.277636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:11.277652 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:11.277725 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:11.278120 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:11.278663 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:11.279493 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:11.279844 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:11.279934 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:11.282881 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35995: BlockingRecv error: recv got EOF from 127.0.98.2:35995 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 18 23:32:11 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:11 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:11.447078 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:11.441871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.3:53315)
0618 23:32:11.442070 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:11.442088 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:11.442193 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:11.442376 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:11.442377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:11.442586 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:11.442647 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:11.442649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:11.443373 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:11.443374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:11.443946 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:11.443948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:11.444344 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:11.444345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:11.444408 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:11.444922 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:11.444927 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:11.447009 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":79,"threads_started":1}
W20260618 23:32:11.447237 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:11.442111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53315 (local address 127.0.98.62:42663)
0618 23:32:11.442197 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:11.442200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:11.442210 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:32:11.442263 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:11.442264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:11.442280 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:11.442357 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:11.442731 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:11.443275 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:11.444036 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:11.444234 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:11.444332 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:11.447138 (+ 2806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53315: BlockingRecv error: recv got EOF from 127.0.98.3:53315 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 18 23:32:12 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:12 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:12.155822 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:12.150641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.1:42575)
0618 23:32:12.150795 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:12.150812 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:12.150864 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:12.151121 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:12.151122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:12.151286 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:12.151337 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:12.151339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:12.152103 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:12.152104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:12.152648 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:12.152650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:12.153026 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:12.153027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:12.153078 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:12.153647 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:12.153652 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:12.155760 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
W20260618 23:32:12.156008 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:12.150787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42575 (local address 127.0.98.62:42663)
0618 23:32:12.150948 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:12.150950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:12.150961 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:32:12.150997 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:12.150998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:12.151009 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:12.151091 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:12.151423 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:12.152000 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:12.152733 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:12.152930 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:12.152987 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:12.155895 (+ 2908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42575: BlockingRecv error: recv got EOF from 127.0.98.1:42575 (error 108)
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
Jun 18 23:32:12 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:12 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:12.288611 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:12.283443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.2:37621)
0618 23:32:12.283704 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:12.283718 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:12.283816 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:12.284025 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:12.284026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:12.284218 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:12.284288 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:12.284290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:12.285017 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:12.285018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:12.285611 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:12.285613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:12.285963 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:12.285964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:12.286012 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:12.286510 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:12.286515 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:12.288531 (+ 2016us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":135,"threads_started":1}
W20260618 23:32:12.288754 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:12.283581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37621 (local address 127.0.98.62:42663)
0618 23:32:12.283635 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:12.283638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:12.283882 (+ 244us) server_negotiation.cc:407] Connection header received
0618 23:32:12.283923 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:12.283924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:12.283938 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:12.283993 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:12.284382 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:12.284923 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:12.285700 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:12.285888 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:12.285942 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:12.288643 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37621: BlockingRecv error: recv got EOF from 127.0.98.2:37621 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 18 23:32:12 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:12 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:12.452906 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:12.447823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.3:41495)
0618 23:32:12.447998 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:12.448009 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:12.448062 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:12.448287 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:12.448288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:12.448472 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:12.448526 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:12.448528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:12.449187 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:12.449189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:12.449756 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:12.449759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:12.450152 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:12.450154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:12.450222 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:12.450758 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:12.450763 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:12.452823 (+ 2060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
W20260618 23:32:12.453029 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:12.448055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41495 (local address 127.0.98.62:42663)
0618 23:32:12.448155 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:12.448157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:12.448168 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:32:12.448198 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:12.448199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:12.448211 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:12.448277 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:12.448608 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:12.449102 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:12.449856 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:12.450058 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:12.450152 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:12.452926 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41495: BlockingRecv error: recv got EOF from 127.0.98.3:41495 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 18 23:32:13 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:13 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:13.163697 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:13.156351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.1:60015)
0618 23:32:13.156503 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:13.156517 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:13.156591 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:13.156877 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:13.156878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:13.157152 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:13.157249 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:13.157251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:13.158528 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:13.158530 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:13.159690 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:13.159699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:13.160427 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:13.160430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:13.160539 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:13.161199 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:13.161207 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:13.163622 (+ 2415us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
W20260618 23:32:13.163903 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:13.156517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60015 (local address 127.0.98.62:42663)
0618 23:32:13.156708 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:13.156711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:13.156722 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:32:13.156749 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:13.156750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:13.156765 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:13.156855 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:13.157357 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:13.158419 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:13.159841 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:13.160264 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:13.160352 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:13.163767 (+ 3415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60015: BlockingRecv error: recv got EOF from 127.0.98.1:60015 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
Jun 18 23:32:13 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:13 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:13.294378 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:13.289274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.2:36155)
0618 23:32:13.289469 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:13.289482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:13.289570 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:13.289735 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:13.289736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:13.289902 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:13.289958 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:13.289960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:13.290732 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:13.290733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:13.291303 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:13.291304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:13.291686 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:13.291687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:13.291736 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:13.292270 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:13.292275 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:13.294314 (+ 2039us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":111,"threads_started":1}
W20260618 23:32:13.294587 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:13.289297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36155 (local address 127.0.98.62:42663)
0618 23:32:13.289381 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:13.289384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:13.289570 (+ 186us) server_negotiation.cc:407] Connection header received
0618 23:32:13.289626 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:13.289627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:13.289640 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:13.289724 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:13.290053 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:13.290620 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:13.291396 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:13.291620 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:13.291685 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:13.294461 (+ 2776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36155: BlockingRecv error: recv got EOF from 127.0.98.2:36155 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 18 23:32:13 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:13 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:13.459760 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:13.453538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.3:50377)
0618 23:32:13.453910 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:13.453927 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:13.454060 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:13.454239 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:13.454240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:13.454468 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:13.454527 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:13.454530 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:13.455318 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:13.455321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:13.456148 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:13.456152 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:13.456816 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:13.456817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:13.456862 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:13.457534 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:13.457540 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:13.459687 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":251,"threads_started":1}
W20260618 23:32:13.460000 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:13.453695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50377 (local address 127.0.98.62:42663)
0618 23:32:13.453880 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:13.453883 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:13.454031 (+ 148us) server_negotiation.cc:407] Connection header received
0618 23:32:13.454148 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:13.454149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:13.454168 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:13.454221 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:13.454615 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:13.455179 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:13.456406 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:13.456729 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:13.456802 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:13.459921 (+ 3119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50377: BlockingRecv error: recv got EOF from 127.0.98.3:50377 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
Jun 18 23:32:14 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:14 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:14.172739 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:14.164359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.1:54705)
0618 23:32:14.164519 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:14.164531 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:14.164593 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:14.164811 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:14.164812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:14.165000 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:14.165067 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:14.165069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:14.165777 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:14.165777 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:14.166371 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:14.166372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:14.166757 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:14.166758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:14.166805 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:14.170287 (+ 3482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:14.170295 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:14.172654 (+ 2359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
W20260618 23:32:14.172909 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:14.164464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54705 (local address 127.0.98.62:42663)
0618 23:32:14.164639 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:14.164641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:14.164653 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:32:14.164681 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:14.164682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:14.164694 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:14.164788 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:14.165175 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:14.165691 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:14.166453 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:14.166673 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:14.166733 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:14.172778 (+ 6045us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54705: BlockingRecv error: recv got EOF from 127.0.98.1:54705 (error 108)
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":113,"threads_started":1}
Jun 18 23:32:14 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:14 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:14.300266 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:14.294981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.2:55325)
0618 23:32:14.295138 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:14.295148 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:14.295201 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:14.295372 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:14.295373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:14.295537 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:14.295641 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:14.295643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:14.296413 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:14.296414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:14.297011 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:14.297013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:14.297424 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:14.297425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:14.297488 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:14.298006 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:14.298012 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:14.300181 (+ 2169us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
W20260618 23:32:14.300405 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:14.295154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55325 (local address 127.0.98.62:42663)
0618 23:32:14.295222 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:14.295224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:14.295244 (+ 20us) server_negotiation.cc:407] Connection header received
0618 23:32:14.295272 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:14.295272 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:14.295284 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:14.295348 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:14.295750 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:14.296305 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:14.297111 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:14.297340 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:14.297408 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:14.300313 (+ 2905us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55325: BlockingRecv error: recv got EOF from 127.0.98.2:55325 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 18 23:32:14 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:14 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:14.465948 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:14.460525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.3:41329)
0618 23:32:14.460728 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:14.460744 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:14.460824 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:14.461142 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:14.461143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:14.461381 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:14.461444 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:14.461447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:14.462183 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:14.462183 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:14.462768 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:14.462770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:14.463165 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:14.463166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:14.463210 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:14.463810 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:14.463815 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:14.465881 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":130,"threads_started":1}
W20260618 23:32:14.466104 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:14.460619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41329 (local address 127.0.98.62:42663)
0618 23:32:14.460674 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:14.460676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:14.460837 (+ 161us) server_negotiation.cc:407] Connection header received
0618 23:32:14.461010 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:14.461011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:14.461032 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:14.461106 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:14.461525 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:14.462070 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:14.462882 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:14.463080 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:14.463150 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:14.466011 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41329: BlockingRecv error: recv got EOF from 127.0.98.3:41329 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 18 23:32:15 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:15 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:15.180158 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:15.173471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.1:54831)
0618 23:32:15.173701 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:15.173715 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:15.173782 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:15.174264 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:15.174265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:15.174544 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:15.174648 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:15.174651 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:15.175919 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:15.175920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:15.176872 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:15.176874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:15.177459 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:15.177459 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:15.177506 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:15.178081 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:15.178085 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:15.180083 (+ 1998us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":89,"threads_started":1}
W20260618 23:32:15.180397 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:15.173716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54831 (local address 127.0.98.62:42663)
0618 23:32:15.174035 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:15.174038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:15.174050 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:32:15.174081 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:15.174082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:15.174098 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:15.174255 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:15.174782 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:15.175805 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:15.176999 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:15.177358 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:15.177446 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:15.180299 (+ 2853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54831: BlockingRecv error: recv got EOF from 127.0.98.1:54831 (error 108)
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":83,"threads_started":1}
Jun 18 23:32:15 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:15 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:15.306841 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:15.300911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.2:50991)
0618 23:32:15.301160 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:15.301176 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:15.301276 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:15.301482 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:15.301484 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:15.301676 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:15.301727 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:15.301729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:15.302417 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:15.302418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:15.302959 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:15.302961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:15.303337 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:15.303339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:15.303400 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:15.304033 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:15.304044 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:15.306726 (+ 2682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":127,"threads_started":1}
W20260618 23:32:15.306998 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:15.301020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50991 (local address 127.0.98.62:42663)
0618 23:32:15.301139 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:15.301142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:15.301255 (+ 113us) server_negotiation.cc:407] Connection header received
0618 23:32:15.301374 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:15.301375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:15.301396 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:15.301476 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:15.301815 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:15.302325 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:15.303052 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:15.303247 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:15.303335 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:15.306896 (+ 3561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50991: BlockingRecv error: recv got EOF from 127.0.98.2:50991 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 18 23:32:15 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:15 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:15.472316 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:15.466834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.3:34523)
0618 23:32:15.467005 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:15.467019 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:15.467106 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:15.467418 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:15.467420 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:15.467667 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:15.467721 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:15.467723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:15.468581 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:15.468582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:15.469103 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:15.469105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:15.469535 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:15.469536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:15.469586 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:15.470203 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:15.470207 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:15.472200 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":116,"threads_started":1}
W20260618 23:32:15.472368 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:15.466974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34523 (local address 127.0.98.62:42663)
0618 23:32:15.467090 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:15.467092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:15.467110 (+ 18us) server_negotiation.cc:407] Connection header received
0618 23:32:15.467199 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:15.467200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:15.467218 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:15.467319 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:15.467840 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:15.468498 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:15.469202 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:15.469455 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:15.469549 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:15.472297 (+ 2748us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34523: BlockingRecv error: recv got EOF from 127.0.98.3:34523 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 18 23:32:16 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:16 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:16.186450 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:16.180738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.1:51499)
0618 23:32:16.180943 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:16.180955 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:16.181018 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:16.181288 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:16.181289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:16.181494 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:16.181562 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:16.181564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:16.182274 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:16.182274 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:16.182852 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:16.182853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:16.183228 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:16.183229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:16.183270 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:16.184081 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:16.184087 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:16.186343 (+ 2256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":122,"threads_started":1}
W20260618 23:32:16.186666 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:16.180937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51499 (local address 127.0.98.62:42663)
0618 23:32:16.181107 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:16.181109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:16.181120 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:32:16.181149 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:16.181150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:16.181174 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:16.181267 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:16.181668 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:16.182192 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:16.182941 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:16.183149 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:16.183266 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:16.186557 (+ 3291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51499: BlockingRecv error: recv got EOF from 127.0.98.1:51499 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":104,"threads_started":1}
Jun 18 23:32:16 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:16 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:16.312711 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:16.307515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.2:47883)
0618 23:32:16.307769 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:16.307783 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:16.307876 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:16.308038 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:16.308039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:16.308195 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:16.308249 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:16.308250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:16.309043 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:16.309044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:16.309725 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:16.309727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:16.310189 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:16.310190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:16.310245 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:16.310763 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:16.310767 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:16.312612 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":166,"threads_started":1}
W20260618 23:32:16.312767 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:16.307673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47883 (local address 127.0.98.62:42663)
0618 23:32:16.307755 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:16.307757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:16.307876 (+ 119us) server_negotiation.cc:407] Connection header received
0618 23:32:16.307937 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:16.307938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:16.307951 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:16.308017 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:16.308344 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:16.308897 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:16.309834 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:16.310095 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:16.310190 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:16.312704 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47883: BlockingRecv error: recv got EOF from 127.0.98.2:47883 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 18 23:32:16 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:16 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:16.477720 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:16.472938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.3:59727)
0618 23:32:16.473103 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:16.473142 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:16.473207 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:16.473369 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:16.473370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:16.473573 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:16.473619 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:16.473622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:16.474305 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:16.474306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:16.474845 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:16.474846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:16.475226 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:16.475227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:16.475289 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:16.475873 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:16.475877 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:16.477619 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":110,"threads_started":1}
W20260618 23:32:16.477762 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:16.473096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59727 (local address 127.0.98.62:42663)
0618 23:32:16.473187 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:16.473189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:16.473201 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:32:16.473277 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:16.473278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:16.473291 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:16.473364 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:16.473704 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:16.474216 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:16.474955 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:16.475142 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:16.475226 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:16.477699 (+ 2473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59727: BlockingRecv error: recv got EOF from 127.0.98.3:59727 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 18 23:32:17 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:17 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:17.193220 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:17.187231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.1:40995)
0618 23:32:17.187424 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:17.187441 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:17.187593 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:17.187976 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:17.187977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:17.188312 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:17.188400 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:17.188404 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:17.189158 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:17.189159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:17.189758 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:17.189760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:17.190143 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:17.190144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:17.190194 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:17.190783 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:17.190788 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:17.193131 (+ 2343us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":71,"threads_started":1}
W20260618 23:32:17.193393 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:17.187514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40995 (local address 127.0.98.62:42663)
0618 23:32:17.187691 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:17.187693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:17.187714 (+ 21us) server_negotiation.cc:407] Connection header received
0618 23:32:17.187762 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:17.187763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:17.187779 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:17.187874 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:17.188508 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:17.189073 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:17.189849 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:17.190056 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:17.190118 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:17.193296 (+ 3178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40995: BlockingRecv error: recv got EOF from 127.0.98.1:40995 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":111,"threads_started":1}
Jun 18 23:32:17 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:17 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:17.319046 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:17.313365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.2:42473)
0618 23:32:17.313631 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:17.313648 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:17.313781 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:17.313983 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:17.313984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:17.314206 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:17.314268 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:17.314270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:17.315179 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:17.315180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:17.315832 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:17.315834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:17.316208 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:17.316210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:17.316265 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:17.316843 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:17.316848 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:17.318975 (+ 2127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":118,"threads_started":1}
W20260618 23:32:17.319243 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:17.313504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42473 (local address 127.0.98.62:42663)
0618 23:32:17.313625 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:17.313628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:17.313753 (+ 125us) server_negotiation.cc:407] Connection header received
0618 23:32:17.313884 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:17.313885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:17.313905 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:17.313975 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:17.314364 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:17.315087 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:17.315921 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:17.316120 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:17.316208 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:17.319154 (+ 2946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42473: BlockingRecv error: recv got EOF from 127.0.98.2:42473 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 18 23:32:17 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:17 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:17.484205 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:17.478577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.3:34173)
0618 23:32:17.478761 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:17.478777 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:17.478877 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:17.479095 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:17.479096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:17.479276 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:17.479331 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:17.479333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:17.480236 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:17.480236 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:17.480793 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:17.480794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:17.481168 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:17.481169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:17.481211 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:17.481743 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:17.481747 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:17.484081 (+ 2334us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":91,"threads_started":1}
W20260618 23:32:17.484301 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:17.478627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34173 (local address 127.0.98.62:42663)
0618 23:32:17.478680 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:17.478682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:17.478868 (+ 186us) server_negotiation.cc:407] Connection header received
0618 23:32:17.478946 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:17.478947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:17.478967 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:17.479044 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:17.479459 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:17.480144 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:17.480883 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:17.481087 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:17.481152 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:17.484180 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34173: BlockingRecv error: recv got EOF from 127.0.98.3:34173 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 18 23:32:18 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:18 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:18.198716 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:18.193772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.1:48709)
0618 23:32:18.193919 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:18.193940 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:18.193997 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:18.194191 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:18.194192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:18.194359 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:18.194415 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:18.194418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:18.195146 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:18.195147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:18.195766 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:18.195768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:18.196203 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:18.196204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:18.196258 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:18.196792 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:18.196796 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:18.198655 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
W20260618 23:32:18.198930 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:18.193872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48709 (local address 127.0.98.62:42663)
0618 23:32:18.194035 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:18.194037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:18.194045 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:32:18.194073 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:18.194074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:18.194086 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:18.194168 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:18.194495 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:18.195062 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:18.195865 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:18.196096 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:18.196171 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:18.198823 (+ 2652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48709: BlockingRecv error: recv got EOF from 127.0.98.1:48709 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
Jun 18 23:32:18 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:18 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:18.325238 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:18.319615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.2:36605)
0618 23:32:18.319815 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:18.319854 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:18.319991 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:18.320196 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:18.320197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:18.320432 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:18.320497 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:18.320500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:18.321262 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:18.321264 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:18.321832 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:18.321834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:18.322207 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:18.322208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:18.322261 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:18.322850 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:18.322858 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:18.325177 (+ 2319us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
W20260618 23:32:18.325479 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:18.319813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36605 (local address 127.0.98.62:42663)
0618 23:32:18.319902 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:18.319905 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:18.319970 (+ 65us) server_negotiation.cc:407] Connection header received
0618 23:32:18.320096 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:18.320097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:18.320117 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:18.320187 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:18.320588 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:18.321161 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:18.321923 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:18.322126 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:18.322209 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:18.325401 (+ 3192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36605: BlockingRecv error: recv got EOF from 127.0.98.2:36605 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 18 23:32:18 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:18 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:18.490279 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:18.484881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.3:51449)
0618 23:32:18.485124 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:18.485144 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:18.485243 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:18.485427 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:18.485428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:18.485619 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:18.485666 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:18.485668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:18.486359 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:18.486360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:18.486877 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:18.486878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:18.487260 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:18.487261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:18.487312 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:18.487939 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:18.487943 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:18.490173 (+ 2230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":129,"threads_started":1}
W20260618 23:32:18.490336 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:18.485000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51449 (local address 127.0.98.62:42663)
0618 23:32:18.485099 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:18.485101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:18.485218 (+ 117us) server_negotiation.cc:407] Connection header received
0618 23:32:18.485333 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:18.485334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:18.485348 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:18.485418 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:18.485749 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:18.486263 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:18.486961 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:18.487179 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:18.487261 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:18.490265 (+ 3004us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51449: BlockingRecv error: recv got EOF from 127.0.98.3:51449 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 18 23:32:19 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:19 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:19.208043 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:19.199334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.1:36397)
0618 23:32:19.199539 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:19.199578 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:19.199678 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:19.199884 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:19.199885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:19.200093 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:19.200163 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:19.200165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:19.200877 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:19.200879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:19.201577 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:19.201579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:19.202010 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:19.202010 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:19.202062 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:19.205710 (+ 3648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:19.205716 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:19.207932 (+ 2216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":66,"threads_started":1}
W20260618 23:32:19.208185 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:19.199532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36397 (local address 127.0.98.62:42663)
0618 23:32:19.199724 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:19.199726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:19.199737 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:32:19.199767 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:19.199767 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:19.199778 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:19.199854 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:19.200246 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:19.200771 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:19.201673 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:19.201917 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:19.202011 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:19.208081 (+ 6070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36397: BlockingRecv error: recv got EOF from 127.0.98.1:36397 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":118,"threads_started":1}
Jun 18 23:32:19 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:19 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:19.331472 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:19.325968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.2:51145)
0618 23:32:19.326150 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:19.326166 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:19.326251 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:19.326467 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:19.326475 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:19.326701 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:19.326774 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:19.326776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:19.327530 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:19.327531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:19.328141 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:19.328143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:19.328527 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:19.328528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:19.328584 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:19.329376 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:19.329382 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:19.331354 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":115,"threads_started":1}
W20260618 23:32:19.331530 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:19.326116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51145 (local address 127.0.98.62:42663)
0618 23:32:19.326194 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:19.326197 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:19.326227 (+ 30us) server_negotiation.cc:407] Connection header received
0618 23:32:19.326354 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:19.326355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:19.326367 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:19.326458 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:19.326861 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:19.327440 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:19.328234 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:19.328443 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:19.328530 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:19.331456 (+ 2926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51145: BlockingRecv error: recv got EOF from 127.0.98.2:51145 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 18 23:32:19 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:19 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:19.496781 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:19.490974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.3:42167)
0618 23:32:19.491140 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:19.491154 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:19.491233 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:19.491494 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:19.491495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:19.491747 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:19.491826 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:19.491828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:19.492847 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:19.492849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:19.493637 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:19.493639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:19.494005 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:19.494007 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:19.494048 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:19.494597 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:19.494601 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:19.496718 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":103,"threads_started":1}
W20260618 23:32:19.496999 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:19.491231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42167 (local address 127.0.98.62:42663)
0618 23:32:19.491330 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:19.491331 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:19.491343 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:32:19.491371 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:19.491372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:19.491384 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:19.491465 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:19.491912 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:19.492718 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:19.493738 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:19.493927 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:19.493990 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:19.496858 (+ 2868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42167: BlockingRecv error: recv got EOF from 127.0.98.3:42167 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 18 23:32:20 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:20 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:20.214424 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:20.208685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.1:52195)
0618 23:32:20.208878 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:20.208897 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:20.209008 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:20.209180 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:20.209181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:20.209387 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:20.209456 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:20.209458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:20.210159 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:20.210160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:20.210822 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:20.210825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:20.211309 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:20.211311 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:20.211364 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:20.211972 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:20.211978 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:20.214322 (+ 2344us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":75,"threads_started":1}
W20260618 23:32:20.214473 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:20.208794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52195 (local address 127.0.98.62:42663)
0618 23:32:20.208992 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:20.208994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:20.209006 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:32:20.209068 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:20.209069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:20.209080 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:20.209151 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:20.209532 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:20.210041 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:20.210933 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:20.211193 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:20.211261 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:20.214401 (+ 3140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52195: BlockingRecv error: recv got EOF from 127.0.98.1:52195 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":109,"threads_started":1}
Jun 18 23:32:20 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:20 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:20.337981 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:20.332260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.2:35627)
0618 23:32:20.332537 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:20.332554 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:20.332687 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:20.332860 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:20.332861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:20.333089 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:20.333151 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:20.333153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:20.334049 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:20.334050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:20.334722 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:20.334724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:20.335128 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:20.335129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:20.335186 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:20.335860 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:20.335865 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:20.337916 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":111,"threads_started":1}
W20260618 23:32:20.338172 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:20.332390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35627 (local address 127.0.98.62:42663)
0618 23:32:20.332525 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:20.332528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:20.332660 (+ 132us) server_negotiation.cc:407] Connection header received
0618 23:32:20.332759 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:20.332760 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:20.332774 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:20.332852 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:20.333243 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:20.333979 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:20.334819 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:20.335038 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:20.335126 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:20.338078 (+ 2952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35627: BlockingRecv error: recv got EOF from 127.0.98.2:35627 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Jun 18 23:32:20 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:20 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:20.502262 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:20.497389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.3:47613)
0618 23:32:20.497554 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:20.497568 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:20.497626 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:20.497838 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:20.497839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:20.498020 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:20.498072 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:20.498074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:20.498828 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:20.498830 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:20.499400 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:20.499402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:20.499839 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:20.499840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:20.499902 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:20.500407 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:20.500412 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:20.502164 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":123,"threads_started":1}
W20260618 23:32:20.502313 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:20.497528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47613 (local address 127.0.98.62:42663)
0618 23:32:20.497603 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:20.497605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:20.497619 (+ 14us) server_negotiation.cc:407] Connection header received
0618 23:32:20.497693 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:20.497694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:20.497711 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:20.497800 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:20.498193 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:20.498733 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:20.499501 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:20.499752 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:20.499839 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:20.502244 (+ 2405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47613: BlockingRecv error: recv got EOF from 127.0.98.3:47613 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 18 23:32:21 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:21 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:21.220347 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:21.215023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.1:60517)
0618 23:32:21.215199 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:21.215212 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:21.215286 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:21.215527 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:21.215528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:21.215765 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:21.215856 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:21.215858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:21.216564 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:21.216565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:21.217165 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:21.217166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:21.217531 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:21.217532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:21.217595 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:21.218127 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:21.218132 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:21.220238 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":110,"threads_started":1}
W20260618 23:32:21.220429 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:21.215094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60517 (local address 127.0.98.62:42663)
0618 23:32:21.215256 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:21.215258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:21.215273 (+ 15us) server_negotiation.cc:407] Connection header received
0618 23:32:21.215359 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:21.215360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:21.215370 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:21.215477 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:21.215946 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:21.216470 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:21.217249 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:21.217435 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:21.217490 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:21.220370 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60517: BlockingRecv error: recv got EOF from 127.0.98.1:60517 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
Jun 18 23:32:21 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:21 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:21.343287 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:21.338588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.2:43001)
0618 23:32:21.338737 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:21.338774 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:21.338848 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:21.339021 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:21.339021 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:21.339200 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:21.339251 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:21.339253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:21.339968 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:21.339969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:21.340500 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:21.340501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:21.340865 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:21.340866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:21.340925 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:21.341445 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:21.341450 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:21.343175 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
W20260618 23:32:21.343321 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:21.338732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43001 (local address 127.0.98.62:42663)
0618 23:32:21.338826 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:21.338829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:21.338842 (+ 13us) server_negotiation.cc:407] Connection header received
0618 23:32:21.338930 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:21.338931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:21.338941 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:21.339002 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:21.339329 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:21.339885 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:21.340594 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:21.340778 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:21.340863 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:21.343256 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43001: BlockingRecv error: recv got EOF from 127.0.98.2:43001 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 18 23:32:21 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:21 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:21.508246 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:21.502883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.3:38301)
0618 23:32:21.503142 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:21.503158 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:21.503277 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:21.503446 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:21.503447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:21.503706 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:21.503770 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:21.503773 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:21.504485 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:21.504486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:21.505083 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:21.505084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:21.505496 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:21.505497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:21.505588 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:21.506245 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:21.506250 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:21.508173 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":94,"threads_started":1}
W20260618 23:32:21.508389 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:21.503053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38301 (local address 127.0.98.62:42663)
0618 23:32:21.503167 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:21.503169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:21.503260 (+ 91us) server_negotiation.cc:407] Connection header received
0618 23:32:21.503347 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:21.503348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:21.503361 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:21.503415 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:21.503847 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:21.504389 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:21.505155 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:21.505413 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:21.505473 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:21.508275 (+ 2802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38301: BlockingRecv error: recv got EOF from 127.0.98.3:38301 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 18 23:32:22 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:22 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:22.226835 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:22.220928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.1:32835)
0618 23:32:22.221112 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:22.221126 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:22.221218 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:22.221393 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:22.221393 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:22.221607 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:22.221687 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:22.221689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:22.222511 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:22.222513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:22.223145 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:22.223147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:22.223540 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:22.223541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:22.223685 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:22.224330 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:22.224335 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:22.226757 (+ 2422us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":99,"threads_started":1}
W20260618 23:32:22.226969 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:22.221063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32835 (local address 127.0.98.62:42663)
0618 23:32:22.221191 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:22.221194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:22.221205 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:32:22.221279 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:22.221280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:22.221290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:22.221357 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:22.221807 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:22.222395 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:22.223242 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:22.223457 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:22.223519 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:22.226871 (+ 3352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:32835: BlockingRecv error: recv got EOF from 127.0.98.1:32835 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
Jun 18 23:32:22 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:22 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:22.348934 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:22.343913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.2:36043)
0618 23:32:22.344157 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:22.344176 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:22.344260 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:22.344441 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:22.344441 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:22.344615 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:22.344669 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:22.344671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:22.345418 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:22.345419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:22.345980 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:22.345982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:22.346351 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:22.346351 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:22.346394 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:22.346922 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:22.346926 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:22.348857 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":101,"threads_started":1}
W20260618 23:32:22.349128 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:22.344150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36043 (local address 127.0.98.62:42663)
0618 23:32:22.344252 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:22.344254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:22.344265 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:32:22.344344 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:22.344345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:22.344357 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:22.344410 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:22.344784 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:22.345333 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:22.346066 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:22.346273 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:22.346354 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:22.349011 (+ 2657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36043: BlockingRecv error: recv got EOF from 127.0.98.2:36043 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 18 23:32:22 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:22 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:22.513962 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:22.508766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.3:38279)
0618 23:32:22.508953 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:22.508967 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:22.509076 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:22.509230 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:22.509231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:22.509435 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:22.509500 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:22.509502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:22.510202 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:22.510203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:22.510808 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:22.510809 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:22.511194 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:22.511195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:22.511277 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:22.511923 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:22.511928 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:22.513879 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":87,"threads_started":1}
W20260618 23:32:22.514118 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:22.508898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38279 (local address 127.0.98.62:42663)
0618 23:32:22.508949 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:22.508952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:22.509078 (+ 126us) server_negotiation.cc:407] Connection header received
0618 23:32:22.509130 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:22.509131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:22.509144 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:22.509202 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:22.509572 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:22.510111 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:22.510894 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:22.511098 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:22.511156 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:22.514014 (+ 2858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38279: BlockingRecv error: recv got EOF from 127.0.98.3:38279 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 18 23:32:23 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:23 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:23.232824 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:23.227461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.1:34609)
0618 23:32:23.227636 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:23.227648 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:23.227713 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:23.227985 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:23.227985 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:23.228185 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:23.228241 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:23.228243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:23.228958 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:23.228958 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:23.229524 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:23.229526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:23.229883 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:23.229884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:23.229924 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:23.230573 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:23.230580 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:23.232696 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":70,"threads_started":1}
W20260618 23:32:23.232865 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:23.227630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34609 (local address 127.0.98.62:42663)
0618 23:32:23.227806 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:23.227809 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:23.227825 (+ 16us) server_negotiation.cc:407] Connection header received
0618 23:32:23.227855 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:23.227856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:23.227869 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:23.227956 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:23.228324 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:23.228881 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:23.229607 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:23.229809 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:23.229871 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:23.232783 (+ 2912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34609: BlockingRecv error: recv got EOF from 127.0.98.1:34609 (error 108)
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":105,"threads_started":1}
Jun 18 23:32:23 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:23 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:23.354823 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:23.349626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.2:42485)
0618 23:32:23.349789 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:23.349803 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:23.349867 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:23.350083 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:23.350084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:23.350243 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:23.350295 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:23.350297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:23.351092 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:23.351093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:23.351786 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:23.351788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:23.352361 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:23.352362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:23.352454 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:23.353015 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:23.353019 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:23.354764 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
W20260618 23:32:23.355121 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:23.349760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42485 (local address 127.0.98.62:42663)
0618 23:32:23.349880 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:23.349882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:23.349896 (+ 14us) server_negotiation.cc:407] Connection header received
0618 23:32:23.349940 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:23.349941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:23.349963 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:23.350053 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:23.350390 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:23.350964 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:23.351893 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:23.352204 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:23.352352 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:23.355043 (+ 2691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42485: BlockingRecv error: recv got EOF from 127.0.98.2:42485 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 18 23:32:23 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:23 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:23.519953 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:23.514658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.3:36999)
0618 23:32:23.514847 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:23.514861 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:23.514937 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:23.515128 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:23.515129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:23.515327 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:23.515406 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:23.515409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:23.516216 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:23.516217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:23.516807 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:23.516809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:23.517160 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:23.517161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:23.517205 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:23.517750 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:23.517757 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:23.519878 (+ 2121us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":111,"threads_started":1}
W20260618 23:32:23.520103 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:23.514804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36999 (local address 127.0.98.62:42663)
0618 23:32:23.514871 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:23.514873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:23.514936 (+ 63us) server_negotiation.cc:407] Connection header received
0618 23:32:23.515022 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:23.515023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:23.515043 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:23.515098 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:23.515507 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:23.516108 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:23.516894 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:23.517087 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:23.517150 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:23.519989 (+ 2839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36999: BlockingRecv error: recv got EOF from 127.0.98.3:36999 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 18 23:32:24 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:24 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:24.242164 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:24.233614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.1:45187)
0618 23:32:24.233812 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:24.233834 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:24.233915 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:24.234129 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:24.234130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:24.234357 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:24.234426 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:24.234429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:24.235306 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:24.235308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:24.236000 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:24.236002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:24.236398 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:24.236399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:24.236448 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:24.239949 (+ 3501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:24.239979 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:24.242088 (+ 2109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":121,"threads_started":1}
W20260618 23:32:24.242313 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:24.233699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45187 (local address 127.0.98.62:42663)
0618 23:32:24.233851 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:24.233853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:24.233895 (+ 42us) server_negotiation.cc:407] Connection header received
0618 23:32:24.233999 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:24.233999 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:24.234012 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:24.234098 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:24.234552 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:24.235189 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:24.236114 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:24.236309 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:24.236387 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:24.242195 (+ 5808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45187: BlockingRecv error: recv got EOF from 127.0.98.1:45187 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
Jun 18 23:32:24 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:32:24 dist-test-slave-6224 krb5kdc[8542](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:32:24.360539 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:24.355506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42663 (local address 127.0.98.2:57757)
0618 23:32:24.355780 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:24.355793 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:24.355899 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:24.356055 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:24.356056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:24.356253 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:24.356319 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:24.356320 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:24.357022 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:24.357023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:24.357597 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:24.357599 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:24.357967 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:24.357968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:24.358030 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:24.358599 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:24.358604 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:24.360432 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:42663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":162,"threads_started":1}
W20260618 23:32:24.360567 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:24.355654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57757 (local address 127.0.98.62:42663)
0618 23:32:24.355753 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:24.355755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:24.355875 (+ 120us) server_negotiation.cc:407] Connection header received
0618 23:32:24.355962 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:24.355963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:24.355974 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:24.356048 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:24.356402 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:24.356922 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:24.357690 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:24.357880 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:24.357966 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:24.360505 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57757: BlockingRecv error: recv got EOF from 127.0.98.2:57757 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 23:32:24.424431 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 9125
I20260618 23:32:24.429643 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 9277
I20260618 23:32:24.434576 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 9409
I20260618 23:32:24.439265 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 9055
2026-06-18T23:32:24Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17577 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-18T23:32:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:32:24Z Disabled control of system clock
I20260618 23:32:24.470129 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36447
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260618 23:32:24.554594 9605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:24.554806 9605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:24.554826 9605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:24.555974 9605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:32:24.556020 9605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:24.556033 9605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:24.556046 9605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:32:24.556061 9605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:32:24.557354 9605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36447
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:24.557554 9605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:24.557798 9605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:24.563100 9613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:24.563254 9610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:24.563268 9611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:24.563342 9605 server_base.cc:1034] running on GCE node
I20260618 23:32:24.563828 9605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:24.564112 9605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:24.565275 9605 hybrid_clock.cc:648] HybridClock initialized: now 1781825544565246 us; error 57 us; skew 500 ppm
I20260618 23:32:24.566635 9605 webserver.cc:460] Webserver started at http://127.0.98.62:38555/ using document root <none> and password file <none>
I20260618 23:32:24.566901 9605 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:24.566953 9605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:24.567106 9605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:32:24.568300 9605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "135ebdefd2e0473a85113013ed312775"
format_stamp: "Formatted at 2026-06-18 23:32:24 on dist-test-slave-6224"
server_key: "dba4b2094a8fa631619d3d788da49342"
server_key_iv: "614229705b95215b2bd20fe38caff16a"
server_key_version: "encryptionkey@0"
I20260618 23:32:24.568660 9605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "135ebdefd2e0473a85113013ed312775"
format_stamp: "Formatted at 2026-06-18 23:32:24 on dist-test-slave-6224"
server_key: "dba4b2094a8fa631619d3d788da49342"
server_key_iv: "614229705b95215b2bd20fe38caff16a"
server_key_version: "encryptionkey@0"
I20260618 23:32:24.570055 9605 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:32:24.570916 9619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:24.571187 9605 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:24.571273 9605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "135ebdefd2e0473a85113013ed312775"
format_stamp: "Formatted at 2026-06-18 23:32:24 on dist-test-slave-6224"
server_key: "dba4b2094a8fa631619d3d788da49342"
server_key_iv: "614229705b95215b2bd20fe38caff16a"
server_key_version: "encryptionkey@0"
I20260618 23:32:24.571368 9605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:24.586951 9605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:24.587298 9605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:24.587442 9605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:24.591969 9605 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36447
I20260618 23:32:24.592000 9671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36447 every 8 connection(s)
I20260618 23:32:24.592365 9605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:32:24.593199 9672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:24.595346 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 9605
I20260618 23:32:24.595458 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:32:24.595716 392 external_mini_cluster.cc:1428] Setting key f18e982360a58c1b4bb71752a78eb968
I20260618 23:32:24.596633 9672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775: Bootstrap starting.
I20260618 23:32:24.597358 9672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:24.597786 9672 log.cc:824] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775: Log is configured to *not* fsync() on all Append() calls
I20260618 23:32:24.598670 9672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775: No bootstrap required, opened a new log
I20260618 23:32:24.601438 9672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135ebdefd2e0473a85113013ed312775" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } }
I20260618 23:32:24.601686 9672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:24.601723 9672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 135ebdefd2e0473a85113013ed312775, State: Initialized, Role: FOLLOWER
I20260618 23:32:24.601908 9672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135ebdefd2e0473a85113013ed312775" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } }
I20260618 23:32:24.602010 9672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:32:24.602037 9672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:32:24.602111 9672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:24.602594 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:24.596351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38018 (local address 127.0.98.62:36447)
0618 23:32:24.596600 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:24.596609 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:24.596638 (+ 29us) server_negotiation.cc:407] Connection header received
0618 23:32:24.597321 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:24.597329 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:24.597395 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:24.597571 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:24.597936 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:24.599027 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:24.600266 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:24.600682 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:24.600787 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:24.601671 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:24.601701 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:24.601701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:32:24.601987 (+ 286us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:24.602051 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:32:24.602068 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:24.602133 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:24.602298 (+ 165us) server_negotiation.cc:299] Negotiation successful
0618 23:32:24.602395 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":112,"threads_started":1}
I20260618 23:32:24.602988 9672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135ebdefd2e0473a85113013ed312775" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } }
I20260618 23:32:24.603117 9672 leader_election.cc:302] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 135ebdefd2e0473a85113013ed312775; no voters:
I20260618 23:32:24.603339 9672 leader_election.cc:288] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:32:24.603415 9677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:32:24.603675 9672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:32:24.603701 9677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775 [term 1 LEADER]: Becoming Leader. State: Replica: 135ebdefd2e0473a85113013ed312775, State: Running, Role: LEADER
I20260618 23:32:24.603806 9677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135ebdefd2e0473a85113013ed312775" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } }
I20260618 23:32:24.604313 9678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "135ebdefd2e0473a85113013ed312775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135ebdefd2e0473a85113013ed312775" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } } }
I20260618 23:32:24.604349 9679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 135ebdefd2e0473a85113013ed312775. Latest consensus state: current_term: 1 leader_uuid: "135ebdefd2e0473a85113013ed312775" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "135ebdefd2e0473a85113013ed312775" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36447 } } }
I20260618 23:32:24.604429 9678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775 [sys.catalog]: This master's current role is: LEADER
I20260618 23:32:24.604434 9679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775 [sys.catalog]: This master's current role is: LEADER
I20260618 23:32:24.604693 9686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:32:24.605229 9686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:32:24.607086 9686 catalog_manager.cc:1347] Generated new cluster ID: 8402031f771649f39bec8e4d6ccef49a
I20260618 23:32:24.607151 9686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:32:24.611256 9686 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:32:24.611924 9686 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:32:24.622816 9686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 135ebdefd2e0473a85113013ed312775: Generated new TSK 0
I20260618 23:32:24.623095 9686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:32:24.627584 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--builtin_ntp_servers=127.0.98.20:35479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260618 23:32:24.723487 9696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:24.723778 9696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:24.723812 9696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:24.724985 9696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:24.725054 9696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:24.725099 9696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:32:24.726467 9696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:24.726732 9696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:24.727036 9696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:24.732386 9701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:24.732424 9702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:24.732467 9704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:24.732681 9696 server_base.cc:1034] running on GCE node
I20260618 23:32:24.733042 9696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:24.733327 9696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:24.734481 9696 hybrid_clock.cc:648] HybridClock initialized: now 1781825544734466 us; error 40 us; skew 500 ppm
I20260618 23:32:24.735826 9696 webserver.cc:460] Webserver started at http://127.0.98.1:36865/ using document root <none> and password file <none>
I20260618 23:32:24.736073 9696 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:24.736140 9696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:24.736310 9696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:32:24.737387 9696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "d3a6d3bc2d8641a495b676a6eda681d9"
format_stamp: "Formatted at 2026-06-18 23:32:24 on dist-test-slave-6224"
server_key: "cd89c4e0a9fa025eaacd28475727da7f"
server_key_iv: "c37a1aea77d0aa5bc1b21a90a1476df7"
server_key_version: "encryptionkey@0"
I20260618 23:32:24.737751 9696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d3a6d3bc2d8641a495b676a6eda681d9"
format_stamp: "Formatted at 2026-06-18 23:32:24 on dist-test-slave-6224"
server_key: "cd89c4e0a9fa025eaacd28475727da7f"
server_key_iv: "c37a1aea77d0aa5bc1b21a90a1476df7"
server_key_version: "encryptionkey@0"
I20260618 23:32:24.739053 9696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:32:24.739902 9710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:24.740193 9696 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:24.740300 9696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "d3a6d3bc2d8641a495b676a6eda681d9"
format_stamp: "Formatted at 2026-06-18 23:32:24 on dist-test-slave-6224"
server_key: "cd89c4e0a9fa025eaacd28475727da7f"
server_key_iv: "c37a1aea77d0aa5bc1b21a90a1476df7"
server_key_version: "encryptionkey@0"
I20260618 23:32:24.740391 9696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:24.750849 9696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:24.751132 9696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:24.751283 9696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:24.751494 9696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:32:24.751927 9696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:32:24.751981 9696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:24.752025 9696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:32:24.752051 9696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:24.759150 9696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39309
I20260618 23:32:24.759184 9823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39309 every 8 connection(s)
I20260618 23:32:24.759635 9696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:32:24.763988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 9696
I20260618 23:32:24.764083 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:32:24.764323 392 external_mini_cluster.cc:1428] Setting key e7a3eeca83d0287480e7026d7d0df055
I20260618 23:32:24.765551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--builtin_ntp_servers=127.0.98.20:35479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260618 23:32:24.765573 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:24.760649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50529 (local address 127.0.98.62:36447)
0618 23:32:24.760709 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:24.760711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:24.761034 (+ 323us) server_negotiation.cc:407] Connection header received
0618 23:32:24.761618 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:24.761619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:24.761638 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:24.761699 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:24.762353 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:24.762878 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:24.763910 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:24.764208 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:24.764280 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:24.765303 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:24.765304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:24.765304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:32:24.765309 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:24.765322 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:32:24.765326 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:24.765379 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:24.765469 (+ 90us) server_negotiation.cc:299] Negotiation successful
0618 23:32:24.765494 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260618 23:32:24.765820 9826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:24.760658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36447 (local address 127.0.98.1:50529)
0618 23:32:24.760893 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:24.760932 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:24.761479 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:24.761769 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:24.761777 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:24.761785 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:24.762254 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:24.762258 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:24.762974 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:24.762975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:24.763815 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:24.763817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:24.764299 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:24.764300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:24.764469 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:24.765148 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:32:24.765149 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:24.765189 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:32:24.765221 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:24.765381 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:24.765386 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:24.765389 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:32:24.765473 (+ 84us) client_negotiation.cc:240] Negotiation successful
0618 23:32:24.765571 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":114,"threads_started":1}
I20260618 23:32:24.766331 9824 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36447
I20260618 23:32:24.766423 9824 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:24.766613 9824 heartbeater.cc:511] Master 127.0.98.62:36447 requested a full tablet report, sending...
I20260618 23:32:24.767091 9636 ts_manager.cc:194] Registered new tserver with Master: d3a6d3bc2d8641a495b676a6eda681d9 (127.0.98.1:39309)
I20260618 23:32:24.767815 9636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:50529
W20260618 23:32:24.859267 9827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:24.859576 9827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:24.859604 9827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:24.861027 9827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:24.861073 9827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:24.861101 9827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:32:24.862620 9827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:24.862941 9827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:24.863314 9827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:24.870095 9832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:24.870111 9833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:24.870117 9835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:24.870553 9827 server_base.cc:1034] running on GCE node
I20260618 23:32:24.870858 9827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:24.871205 9827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:24.872376 9827 hybrid_clock.cc:648] HybridClock initialized: now 1781825544872341 us; error 70 us; skew 500 ppm
I20260618 23:32:24.873871 9827 webserver.cc:460] Webserver started at http://127.0.98.2:43005/ using document root <none> and password file <none>
I20260618 23:32:24.874140 9827 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:24.874215 9827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:24.874378 9827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:32:24.875698 9827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "e1157edfd7dd45438d0f4ed5017c9b6a"
format_stamp: "Formatted at 2026-06-18 23:32:24 on dist-test-slave-6224"
server_key: "9a0056f5a94a31218473d814f1b0903e"
server_key_iv: "591fc9efcb7bf3baab821dc23b66dfaa"
server_key_version: "encryptionkey@0"
I20260618 23:32:24.876066 9827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e1157edfd7dd45438d0f4ed5017c9b6a"
format_stamp: "Formatted at 2026-06-18 23:32:24 on dist-test-slave-6224"
server_key: "9a0056f5a94a31218473d814f1b0903e"
server_key_iv: "591fc9efcb7bf3baab821dc23b66dfaa"
server_key_version: "encryptionkey@0"
I20260618 23:32:24.877532 9827 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260618 23:32:24.878388 9841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:24.878646 9827 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260618 23:32:24.878756 9827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "e1157edfd7dd45438d0f4ed5017c9b6a"
format_stamp: "Formatted at 2026-06-18 23:32:24 on dist-test-slave-6224"
server_key: "9a0056f5a94a31218473d814f1b0903e"
server_key_iv: "591fc9efcb7bf3baab821dc23b66dfaa"
server_key_version: "encryptionkey@0"
I20260618 23:32:24.878837 9827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:24.892025 9827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:24.892367 9827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:24.892521 9827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:24.892766 9827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:32:24.893118 9827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:32:24.893182 9827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:24.893222 9827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:32:24.893250 9827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:24.899909 9827 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46103
I20260618 23:32:24.899959 9954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46103 every 8 connection(s)
I20260618 23:32:24.900316 9827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:32:24.902211 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 9827
I20260618 23:32:24.902304 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:32:24.902472 392 external_mini_cluster.cc:1428] Setting key b02a7cdf83601b0bae59f23edb9aba14
I20260618 23:32:24.903630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--builtin_ntp_servers=127.0.98.20:35479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260618 23:32:24.906984 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:24.901177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45253 (local address 127.0.98.62:36447)
0618 23:32:24.901248 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:24.901250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:24.901546 (+ 296us) server_negotiation.cc:407] Connection header received
0618 23:32:24.902177 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:24.902179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:24.902195 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:24.902245 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:24.903253 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:24.904082 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:24.905260 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:24.905487 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:24.905602 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:24.906614 (+ 1012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:24.906616 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:24.906616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:32:24.906621 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:24.906644 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:32:24.906648 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:24.906712 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:24.906819 (+ 107us) server_negotiation.cc:299] Negotiation successful
0618 23:32:24.906853 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260618 23:32:24.907135 9957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:24.901154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36447 (local address 127.0.98.2:45253)
0618 23:32:24.901388 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:24.901435 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:24.902052 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:24.902411 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:24.902421 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:24.902431 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:24.903127 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:24.903133 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:24.904190 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:24.904191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:24.905159 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:24.905162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:24.905589 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:24.905590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:24.905742 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:24.906429 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:32:24.906430 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:24.906468 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:32:24.906501 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:24.906725 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:24.906729 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:24.906733 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:32:24.906796 (+ 63us) client_negotiation.cc:240] Negotiation successful
0618 23:32:24.906916 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":121,"threads_started":1}
I20260618 23:32:24.907759 9955 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36447
I20260618 23:32:24.907887 9955 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:24.908131 9955 heartbeater.cc:511] Master 127.0.98.62:36447 requested a full tablet report, sending...
I20260618 23:32:24.908607 9636 ts_manager.cc:194] Registered new tserver with Master: e1157edfd7dd45438d0f4ed5017c9b6a (127.0.98.2:46103)
I20260618 23:32:24.909200 9636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:45253
W20260618 23:32:24.993705 9958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:24.993965 9958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:24.993993 9958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:24.995154 9958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:24.995193 9958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:24.995218 9958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:32:24.996548 9958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:24.996780 9958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:24.997108 9958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:25.002797 9963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:25.002797 9966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:25.002866 9964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:25.002820 9958 server_base.cc:1034] running on GCE node
I20260618 23:32:25.003361 9958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:25.003732 9958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:25.004897 9958 hybrid_clock.cc:648] HybridClock initialized: now 1781825545004880 us; error 51 us; skew 500 ppm
I20260618 23:32:25.006641 9958 webserver.cc:460] Webserver started at http://127.0.98.3:35239/ using document root <none> and password file <none>
I20260618 23:32:25.006953 9958 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:25.007035 9958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:25.007225 9958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:32:25.008638 9958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "02b43d14035d42d588211add10641809"
format_stamp: "Formatted at 2026-06-18 23:32:25 on dist-test-slave-6224"
server_key: "efe217c89deb197038f6413c8d9fee54"
server_key_iv: "1bb8ce224cfb6985fdb942a2c124db43"
server_key_version: "encryptionkey@0"
I20260618 23:32:25.009078 9958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "02b43d14035d42d588211add10641809"
format_stamp: "Formatted at 2026-06-18 23:32:25 on dist-test-slave-6224"
server_key: "efe217c89deb197038f6413c8d9fee54"
server_key_iv: "1bb8ce224cfb6985fdb942a2c124db43"
server_key_version: "encryptionkey@0"
I20260618 23:32:25.011016 9958 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.002s
I20260618 23:32:25.012243 9972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:25.012529 9958 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:25.012663 9958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "02b43d14035d42d588211add10641809"
format_stamp: "Formatted at 2026-06-18 23:32:25 on dist-test-slave-6224"
server_key: "efe217c89deb197038f6413c8d9fee54"
server_key_iv: "1bb8ce224cfb6985fdb942a2c124db43"
server_key_version: "encryptionkey@0"
I20260618 23:32:25.012758 9958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:25.037037 9958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:25.037458 9958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:25.037626 9958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:25.037914 9958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:32:25.038312 9958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:32:25.038374 9958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:25.038419 9958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:32:25.038457 9958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:25.045917 9958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45923
I20260618 23:32:25.045969 10085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45923 every 8 connection(s)
I20260618 23:32:25.046346 9958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:32:25.049961 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 9958
I20260618 23:32:25.050079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:32:25.050280 392 external_mini_cluster.cc:1428] Setting key c5c83de2b7c1335a12dc6b16a7b5c47e
I20260618 23:32:25.052305 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:25.047213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56173 (local address 127.0.98.62:36447)
0618 23:32:25.047296 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:25.047298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:25.047613 (+ 315us) server_negotiation.cc:407] Connection header received
0618 23:32:25.048222 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:25.048223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:25.048235 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:25.048295 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:25.049017 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:25.049514 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:25.050720 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:25.050975 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:25.051047 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:25.052090 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:25.052091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:25.052092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:32:25.052096 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:25.052110 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:32:25.052114 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:25.052164 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:25.052225 (+ 61us) server_negotiation.cc:299] Negotiation successful
0618 23:32:25.052245 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 23:32:25.052582 10088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:25.047194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36447 (local address 127.0.98.3:56173)
0618 23:32:25.047445 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:25.047486 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:25.048117 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:25.048372 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:25.048376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:25.048385 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:25.048932 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:25.048937 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:25.049605 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:25.049606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:25.050573 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:25.050576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:25.051065 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:25.051066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:25.051255 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:25.051955 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:32:25.051955 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:25.051988 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:32:25.052012 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:25.052166 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:25.052169 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:25.052172 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:32:25.052225 (+ 53us) client_negotiation.cc:240] Negotiation successful
0618 23:32:25.052315 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":113,"threads_started":1}
I20260618 23:32:25.053099 10086 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36447
I20260618 23:32:25.053231 10086 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:25.053447 10086 heartbeater.cc:511] Master 127.0.98.62:36447 requested a full tablet report, sending...
I20260618 23:32:25.053869 9636 ts_manager.cc:194] Registered new tserver with Master: 02b43d14035d42d588211add10641809 (127.0.98.3:45923)
I20260618 23:32:25.054255 9636 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:56173
I20260618 23:32:25.062670 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:32:25.064996 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 9696
W20260618 23:32:25.065039 9675 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:25.064240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38034 (local address 127.0.98.62:36447)
0618 23:32:25.064359 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:25.064361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:25.064569 (+ 208us) server_negotiation.cc:407] Connection header received
0618 23:32:25.064602 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:25.064603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:25.064617 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:25.064686 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:25.064901 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38034: BlockingRecv error: recv got EOF from 127.0.0.1:38034 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260618 23:32:25.069974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 9827
I20260618 23:32:25.075623 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 9958
I20260618 23:32:25.080998 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 9605
2026-06-18T23:32:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (640 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-18T23:32:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:32:25Z Disabled control of system clock
I20260618 23:32:25.111025 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45223
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260618 23:32:25.216743 10097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:25.217113 10097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:25.217168 10097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:25.219153 10097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:32:25.219213 10097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:25.219231 10097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:25.219256 10097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:32:25.219307 10097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:32:25.221597 10097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45223
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:25.221920 10097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:25.222354 10097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:25.228070 10105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:25.228096 10103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:25.228421 10102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:25.228720 10097 server_base.cc:1034] running on GCE node
I20260618 23:32:25.229004 10097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:25.229391 10097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:25.230566 10097 hybrid_clock.cc:648] HybridClock initialized: now 1781825545230535 us; error 54 us; skew 500 ppm
I20260618 23:32:25.232126 10097 webserver.cc:460] Webserver started at http://127.0.98.62:42407/ using document root <none> and password file <none>
I20260618 23:32:25.232409 10097 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:25.232481 10097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:25.232685 10097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:32:25.233840 10097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "2cdb3f49d5294dfdbaf20f756c96cc59"
format_stamp: "Formatted at 2026-06-18 23:32:25 on dist-test-slave-6224"
server_key: "bc3a00a70fce116d305426125ea08330"
server_key_iv: "3a800dd295d28bf84554287f847ba185"
server_key_version: "encryptionkey@0"
I20260618 23:32:25.234277 10097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "2cdb3f49d5294dfdbaf20f756c96cc59"
format_stamp: "Formatted at 2026-06-18 23:32:25 on dist-test-slave-6224"
server_key: "bc3a00a70fce116d305426125ea08330"
server_key_iv: "3a800dd295d28bf84554287f847ba185"
server_key_version: "encryptionkey@0"
I20260618 23:32:25.235836 10097 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260618 23:32:25.236699 10111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:25.236912 10097 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:25.237035 10097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "2cdb3f49d5294dfdbaf20f756c96cc59"
format_stamp: "Formatted at 2026-06-18 23:32:25 on dist-test-slave-6224"
server_key: "bc3a00a70fce116d305426125ea08330"
server_key_iv: "3a800dd295d28bf84554287f847ba185"
server_key_version: "encryptionkey@0"
I20260618 23:32:25.237121 10097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:25.240494 10097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:25.240782 10097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:25.240927 10097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:25.245082 10097 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45223
I20260618 23:32:25.245188 10163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45223 every 8 connection(s)
I20260618 23:32:25.245505 10097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:32:25.246383 10164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:25.246551 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 10097
I20260618 23:32:25.246680 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:32:25.246884 392 external_mini_cluster.cc:1428] Setting key 96102a8d25e43b471a7e0c38748aa91a
I20260618 23:32:25.250219 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:25.247791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44588 (local address 127.0.98.62:45223)
0618 23:32:25.248039 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:25.248052 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:25.248091 (+ 39us) server_negotiation.cc:407] Connection header received
0618 23:32:25.248604 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:25.248615 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:25.248695 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:25.248895 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:25.249582 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:25.249586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:25.249587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:32:25.249592 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:25.249649 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:32:25.249664 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:25.249755 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:25.249799 (+ 44us) server_negotiation.cc:299] Negotiation successful
0618 23:32:25.250004 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
I20260618 23:32:25.251119 10164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59: Bootstrap starting.
I20260618 23:32:25.251878 10164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:25.252137 10164 log.cc:824] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59: Log is configured to *not* fsync() on all Append() calls
I20260618 23:32:25.252928 10164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59: No bootstrap required, opened a new log
I20260618 23:32:25.254473 10164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdb3f49d5294dfdbaf20f756c96cc59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45223 } }
I20260618 23:32:25.254626 10164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:25.254652 10164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cdb3f49d5294dfdbaf20f756c96cc59, State: Initialized, Role: FOLLOWER
I20260618 23:32:25.254776 10164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdb3f49d5294dfdbaf20f756c96cc59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45223 } }
I20260618 23:32:25.254873 10164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:32:25.254896 10164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:32:25.254927 10164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:25.255609 10164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdb3f49d5294dfdbaf20f756c96cc59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45223 } }
I20260618 23:32:25.255744 10164 leader_election.cc:302] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2cdb3f49d5294dfdbaf20f756c96cc59; no voters:
I20260618 23:32:25.255966 10164 leader_election.cc:288] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:32:25.256291 10169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:32:25.256304 10164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:32:25.256544 10169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59 [term 1 LEADER]: Becoming Leader. State: Replica: 2cdb3f49d5294dfdbaf20f756c96cc59, State: Running, Role: LEADER
I20260618 23:32:25.256698 10169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdb3f49d5294dfdbaf20f756c96cc59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45223 } }
I20260618 23:32:25.257175 10169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cdb3f49d5294dfdbaf20f756c96cc59. Latest consensus state: current_term: 1 leader_uuid: "2cdb3f49d5294dfdbaf20f756c96cc59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdb3f49d5294dfdbaf20f756c96cc59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45223 } } }
I20260618 23:32:25.257275 10169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59 [sys.catalog]: This master's current role is: LEADER
I20260618 23:32:25.257488 10170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2cdb3f49d5294dfdbaf20f756c96cc59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2cdb3f49d5294dfdbaf20f756c96cc59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45223 } } }
I20260618 23:32:25.257573 10170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59 [sys.catalog]: This master's current role is: LEADER
I20260618 23:32:25.257987 10178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:32:25.258561 10178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:32:25.260614 10178 catalog_manager.cc:1347] Generated new cluster ID: 569fea42e0144460bc4d8f301f8b31a2
I20260618 23:32:25.260684 10178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:32:25.282760 10178 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:32:25.283002 10178 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:32:25.305717 10178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cdb3f49d5294dfdbaf20f756c96cc59: Generated new TSK 0
I20260618 23:32:25.305979 10178 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:32:25.311442 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45223
--builtin_ntp_servers=127.0.98.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260618 23:32:25.401777 10188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:25.402064 10188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:25.402103 10188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:25.403330 10188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:25.403398 10188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:25.403450 10188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:32:25.404922 10188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:25.405223 10188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:25.405556 10188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:25.410989 10194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:25.411036 10193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:25.411201 10196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:25.411254 10188 server_base.cc:1034] running on GCE node
I20260618 23:32:25.411685 10188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:25.411990 10188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:25.413172 10188 hybrid_clock.cc:648] HybridClock initialized: now 1781825545413147 us; error 48 us; skew 500 ppm
I20260618 23:32:25.414630 10188 webserver.cc:460] Webserver started at http://127.0.98.1:40253/ using document root <none> and password file <none>
I20260618 23:32:25.414914 10188 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:25.414987 10188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:25.415158 10188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:32:25.416435 10188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "2b621df246ff4df3900482724c120ea8"
format_stamp: "Formatted at 2026-06-18 23:32:25 on dist-test-slave-6224"
server_key: "4170aebc4cad4f5fbfa075ca4b66ac74"
server_key_iv: "7e0666735b3adb4fee20fd325eba6e98"
server_key_version: "encryptionkey@0"
I20260618 23:32:25.416842 10188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2b621df246ff4df3900482724c120ea8"
format_stamp: "Formatted at 2026-06-18 23:32:25 on dist-test-slave-6224"
server_key: "4170aebc4cad4f5fbfa075ca4b66ac74"
server_key_iv: "7e0666735b3adb4fee20fd325eba6e98"
server_key_version: "encryptionkey@0"
I20260618 23:32:25.418365 10188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:32:25.419234 10202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:25.419454 10188 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:25.419595 10188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "2b621df246ff4df3900482724c120ea8"
format_stamp: "Formatted at 2026-06-18 23:32:25 on dist-test-slave-6224"
server_key: "4170aebc4cad4f5fbfa075ca4b66ac74"
server_key_iv: "7e0666735b3adb4fee20fd325eba6e98"
server_key_version: "encryptionkey@0"
I20260618 23:32:25.419710 10188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:25.423290 10188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:25.423679 10188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:25.423836 10188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:25.424062 10188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:32:25.424474 10188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:32:25.424535 10188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:25.424575 10188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:32:25.424602 10188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:25.432415 10188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44229
I20260618 23:32:25.432466 10315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44229 every 8 connection(s)
I20260618 23:32:25.432931 10188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:32:25.436681 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:25.433922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38093 (local address 127.0.98.62:45223)
0618 23:32:25.433985 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:25.433988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:25.434299 (+ 311us) server_negotiation.cc:407] Connection header received
0618 23:32:25.434955 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:25.434956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:25.434961 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:25.435046 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:25.436410 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:25.436411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:25.436412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:32:25.436415 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:25.436432 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:32:25.436436 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:25.436496 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:25.436601 (+ 105us) server_negotiation.cc:299] Negotiation successful
0618 23:32:25.436614 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 23:32:25.436937 10318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:25.433904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45223 (local address 127.0.98.1:38093)
0618 23:32:25.434130 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:25.434180 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:25.434849 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:25.435111 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:25.435115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:25.435125 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:25.436220 (+ 1095us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:32:25.436220 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:25.436254 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:32:25.436295 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:25.436519 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:25.436523 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:25.436527 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:32:25.436598 (+ 71us) client_negotiation.cc:240] Negotiation successful
0618 23:32:25.436661 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":113,"threads_started":1}
I20260618 23:32:25.437268 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 10188
I20260618 23:32:25.437389 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:32:25.437616 392 external_mini_cluster.cc:1428] Setting key 6b5a849666876575958a5fe0614c865e
I20260618 23:32:25.437698 10316 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45223
I20260618 23:32:25.437824 10316 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:25.438036 10316 heartbeater.cc:511] Master 127.0.98.62:45223 requested a full tablet report, sending...
I20260618 23:32:25.438711 10128 ts_manager.cc:194] Registered new tserver with Master: 2b621df246ff4df3900482724c120ea8 (127.0.98.1:44229)
I20260618 23:32:25.438849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45223
--builtin_ntp_servers=127.0.98.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260618 23:32:25.528672 10319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:25.528952 10319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:25.528990 10319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:25.530839 10319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:25.530900 10319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:25.530930 10319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:32:25.532500 10319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:25.532796 10319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:25.533152 10319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:25.538671 10327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:25.538686 10324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:25.538690 10325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:25.539186 10319 server_base.cc:1034] running on GCE node
I20260618 23:32:25.539444 10319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:25.539736 10319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:25.540889 10319 hybrid_clock.cc:648] HybridClock initialized: now 1781825545540863 us; error 47 us; skew 500 ppm
I20260618 23:32:25.542358 10319 webserver.cc:460] Webserver started at http://127.0.98.2:37993/ using document root <none> and password file <none>
I20260618 23:32:25.542654 10319 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:25.542717 10319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:25.542912 10319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:32:25.544165 10319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "74106e94d9044229ba0da705f20b4ec6"
format_stamp: "Formatted at 2026-06-18 23:32:25 on dist-test-slave-6224"
server_key: "1d979d7203ed7142a89a526002f4d12d"
server_key_iv: "d085b59f107d6bb27cdf658ca8891ade"
server_key_version: "encryptionkey@0"
I20260618 23:32:25.544556 10319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "74106e94d9044229ba0da705f20b4ec6"
format_stamp: "Formatted at 2026-06-18 23:32:25 on dist-test-slave-6224"
server_key: "1d979d7203ed7142a89a526002f4d12d"
server_key_iv: "d085b59f107d6bb27cdf658ca8891ade"
server_key_version: "encryptionkey@0"
I20260618 23:32:25.545994 10319 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:32:25.546900 10333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:25.547199 10319 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:25.547298 10319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "74106e94d9044229ba0da705f20b4ec6"
format_stamp: "Formatted at 2026-06-18 23:32:25 on dist-test-slave-6224"
server_key: "1d979d7203ed7142a89a526002f4d12d"
server_key_iv: "d085b59f107d6bb27cdf658ca8891ade"
server_key_version: "encryptionkey@0"
I20260618 23:32:25.547386 10319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:25.551000 10319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:25.551297 10319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:25.551445 10319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:25.551725 10319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:32:25.552074 10319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:32:25.552131 10319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:25.552165 10319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:32:25.552191 10319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:25.559343 10319 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36133
I20260618 23:32:25.559464 10446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36133 every 8 connection(s)
I20260618 23:32:25.559836 10319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:32:25.563290 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:25.560749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52667 (local address 127.0.98.62:45223)
0618 23:32:25.560820 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:25.560823 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:25.561058 (+ 235us) server_negotiation.cc:407] Connection header received
0618 23:32:25.561536 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:25.561537 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:25.561544 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:25.561629 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:25.562897 (+ 1268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:25.562899 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:25.562899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:32:25.562903 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:25.562921 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:32:25.562925 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:25.562995 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:25.563196 (+ 201us) server_negotiation.cc:299] Negotiation successful
0618 23:32:25.563213 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260618 23:32:25.563397 10449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:25.560706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45223 (local address 127.0.98.2:52667)
0618 23:32:25.560912 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:25.560954 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:25.561432 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:25.561696 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:25.561700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:25.561708 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:25.562714 (+ 1006us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:32:25.562714 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:25.562742 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:32:25.562770 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:25.563050 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:25.563057 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:25.563059 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:32:25.563169 (+ 110us) client_negotiation.cc:240] Negotiation successful
0618 23:32:25.563222 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":102,"threads_started":1}
I20260618 23:32:25.564131 10447 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45223
I20260618 23:32:25.564277 10447 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:25.564488 10447 heartbeater.cc:511] Master 127.0.98.62:45223 requested a full tablet report, sending...
I20260618 23:32:25.564539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 10319
I20260618 23:32:25.564648 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:32:25.564888 392 external_mini_cluster.cc:1428] Setting key 37bdb75829c75b6882b0784a28defb07
I20260618 23:32:25.564985 10128 ts_manager.cc:194] Registered new tserver with Master: 74106e94d9044229ba0da705f20b4ec6 (127.0.98.2:36133)
I20260618 23:32:25.566066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45223
--builtin_ntp_servers=127.0.98.20:43683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260618 23:32:25.651785 10450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:25.652004 10450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:25.652029 10450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:25.653198 10450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:25.653244 10450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:25.653270 10450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:32:25.654600 10450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:25.654848 10450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:25.655179 10450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:25.660591 10456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:25.660810 10450 server_base.cc:1034] running on GCE node
W20260618 23:32:25.660588 10458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:25.660898 10455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:25.661239 10450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:25.661576 10450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:25.662755 10450 hybrid_clock.cc:648] HybridClock initialized: now 1781825545662747 us; error 51 us; skew 500 ppm
I20260618 23:32:25.664340 10450 webserver.cc:460] Webserver started at http://127.0.98.3:36249/ using document root <none> and password file <none>
I20260618 23:32:25.664628 10450 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:25.664705 10450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:25.664872 10450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:32:25.666095 10450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "f2980ca8a3f94a71994023ee89bfbacb"
format_stamp: "Formatted at 2026-06-18 23:32:25 on dist-test-slave-6224"
server_key: "0e4f5b2edb785fdd48db8137cf55d734"
server_key_iv: "862d1a69e35f0ab5677d986b2fcc3d87"
server_key_version: "encryptionkey@0"
I20260618 23:32:25.666507 10450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f2980ca8a3f94a71994023ee89bfbacb"
format_stamp: "Formatted at 2026-06-18 23:32:25 on dist-test-slave-6224"
server_key: "0e4f5b2edb785fdd48db8137cf55d734"
server_key_iv: "862d1a69e35f0ab5677d986b2fcc3d87"
server_key_version: "encryptionkey@0"
I20260618 23:32:25.668185 10450 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:32:25.669224 10464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:25.669462 10450 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:32:25.669598 10450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "f2980ca8a3f94a71994023ee89bfbacb"
format_stamp: "Formatted at 2026-06-18 23:32:25 on dist-test-slave-6224"
server_key: "0e4f5b2edb785fdd48db8137cf55d734"
server_key_iv: "862d1a69e35f0ab5677d986b2fcc3d87"
server_key_version: "encryptionkey@0"
I20260618 23:32:25.669699 10450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:25.674094 10450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:25.674456 10450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:25.674625 10450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:25.674901 10450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:32:25.675289 10450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:32:25.675350 10450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:25.675400 10450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:32:25.675428 10450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:25.683152 10450 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39267
I20260618 23:32:25.683214 10577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39267 every 8 connection(s)
I20260618 23:32:25.683732 10450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:32:25.687469 10167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:25.684598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41007 (local address 127.0.98.62:45223)
0618 23:32:25.684660 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:25.684662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:25.685014 (+ 352us) server_negotiation.cc:407] Connection header received
0618 23:32:25.685675 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:25.685677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:25.685684 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:25.685743 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:25.687133 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:25.687135 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:25.687136 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:32:25.687140 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:25.687159 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:32:25.687164 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:25.687243 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:25.687362 (+ 119us) server_negotiation.cc:299] Negotiation successful
0618 23:32:25.687382 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260618 23:32:25.687726 10580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:25.684572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45223 (local address 127.0.98.3:41007)
0618 23:32:25.684844 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:25.684886 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:25.685540 (+ 654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:25.685869 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:25.685873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:25.685884 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:25.686948 (+ 1064us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:32:25.686948 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:25.686982 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:32:25.687016 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:25.687255 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:25.687261 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:25.687264 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:32:25.687374 (+ 110us) client_negotiation.cc:240] Negotiation successful
0618 23:32:25.687449 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":124,"threads_started":1}
I20260618 23:32:25.688427 10578 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45223
I20260618 23:32:25.688553 10578 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:25.688747 10578 heartbeater.cc:511] Master 127.0.98.62:45223 requested a full tablet report, sending...
I20260618 23:32:25.689199 10128 ts_manager.cc:194] Registered new tserver with Master: f2980ca8a3f94a71994023ee89bfbacb (127.0.98.3:39267)
I20260618 23:32:25.691835 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 10450
I20260618 23:32:25.691928 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:32:25.692097 392 external_mini_cluster.cc:1428] Setting key 24657104f15275f762f1ab1de57ffd1e
I20260618 23:32:25.693183 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260618 23:32:25.695405 10167 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:32:25.694789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44598 (local address 127.0.98.62:45223)
0618 23:32:25.694901 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:25.694903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:25.695031 (+ 128us) server_negotiation.cc:407] Connection header received
0618 23:32:25.695099 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:25.695099 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:25.695106 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:25.695183 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:25.695323 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44598: BlockingRecv error: recv got EOF from 127.0.0.1:44598 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260618 23:32:25.695591 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 10188
I20260618 23:32:25.704084 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 10319
I20260618 23:32:25.710171 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 10450
I20260618 23:32:25.715438 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 10097
2026-06-18T23:32:25Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (634 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:32:25 dist-test-slave-6224 krb5kdc[10588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:32:25 dist-test-slave-6224 krb5kdc[10588](info): set up 2 sockets
Jun 18 23:32:25 dist-test-slave-6224 krb5kdc[10588](info): commencing operation
krb5kdc: starting...
W20260618 23:32:26.756829 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:32:26 dist-test-slave-6224 krb5kdc[10588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:32:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:32:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:32:26.886404 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37961
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37961
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:32:26.969484 10601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:26.969753 10601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:26.969790 10601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:26.970981 10601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:32:26.971074 10601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:26.971104 10601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:26.971138 10601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:32:26.971166 10601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:32:26.972829 10601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37961
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:26.973130 10601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:26.973371 10601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:26.978621 10606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:26.978649 10609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:26.978672 10607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:26.978865 10601 server_base.cc:1034] running on GCE node
I20260618 23:32:26.979247 10601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:26.979645 10601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:26.980826 10601 hybrid_clock.cc:648] HybridClock initialized: now 1781825546980791 us; error 58 us; skew 500 ppm
Jun 18 23:32:26 dist-test-slave-6224 krb5kdc[10588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:32:26.983924 10601 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:32:26.984542 10601 webserver.cc:460] Webserver started at http://127.0.98.62:33077/ using document root <none> and password file <none>
I20260618 23:32:26.984817 10601 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:26.984911 10601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:26.985067 10601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:32:26.986307 10601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "b204eb5c1ed64308891e97fb857eb90b"
format_stamp: "Formatted at 2026-06-18 23:32:26 on dist-test-slave-6224"
server_key: "5a8a942accaccb4aa18fcd18a825c533"
server_key_iv: "8120f037912569da156c455a7344a387"
server_key_version: "encryptionkey@0"
I20260618 23:32:26.986721 10601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "b204eb5c1ed64308891e97fb857eb90b"
format_stamp: "Formatted at 2026-06-18 23:32:26 on dist-test-slave-6224"
server_key: "5a8a942accaccb4aa18fcd18a825c533"
server_key_iv: "8120f037912569da156c455a7344a387"
server_key_version: "encryptionkey@0"
I20260618 23:32:26.988384 10601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260618 23:32:26.989321 10616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:26.989585 10601 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:26.989699 10601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "b204eb5c1ed64308891e97fb857eb90b"
format_stamp: "Formatted at 2026-06-18 23:32:26 on dist-test-slave-6224"
server_key: "5a8a942accaccb4aa18fcd18a825c533"
server_key_iv: "8120f037912569da156c455a7344a387"
server_key_version: "encryptionkey@0"
I20260618 23:32:26.989796 10601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:27.004120 10601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:27.006708 10601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:27.006892 10601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:27.011999 10601 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37961
I20260618 23:32:27.012051 10668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37961 every 8 connection(s)
I20260618 23:32:27.012557 10601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:32:27.013854 10669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:27.016356 10669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b: Bootstrap starting.
I20260618 23:32:27.016916 10669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:27.017180 10669 log.cc:824] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b: Log is configured to *not* fsync() on all Append() calls
I20260618 23:32:27.018023 10669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b: No bootstrap required, opened a new log
I20260618 23:32:27.019769 10669 raft_consensus.cc:348] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b204eb5c1ed64308891e97fb857eb90b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37961 } }
I20260618 23:32:27.019919 10669 raft_consensus.cc:374] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:27.019945 10669 raft_consensus.cc:729] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b204eb5c1ed64308891e97fb857eb90b, State: Initialized, Role: FOLLOWER
I20260618 23:32:27.020094 10669 consensus_queue.cc:260] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b204eb5c1ed64308891e97fb857eb90b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37961 } }
I20260618 23:32:27.020182 10669 raft_consensus.cc:388] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:32:27.020205 10669 raft_consensus.cc:482] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:32:27.020237 10669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:27.021025 10669 raft_consensus.cc:504] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b204eb5c1ed64308891e97fb857eb90b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37961 } }
I20260618 23:32:27.021122 10669 leader_election.cc:302] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b204eb5c1ed64308891e97fb857eb90b; no voters:
I20260618 23:32:27.021291 10669 leader_election.cc:288] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:32:27.021320 10672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:32:27.021440 10672 raft_consensus.cc:686] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b [term 1 LEADER]: Becoming Leader. State: Replica: b204eb5c1ed64308891e97fb857eb90b, State: Running, Role: LEADER
I20260618 23:32:27.021587 10669 sys_catalog.cc:564] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:32:27.021854 10672 consensus_queue.cc:237] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b204eb5c1ed64308891e97fb857eb90b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37961 } }
I20260618 23:32:27.022372 10673 sys_catalog.cc:455] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b204eb5c1ed64308891e97fb857eb90b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b204eb5c1ed64308891e97fb857eb90b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37961 } } }
I20260618 23:32:27.022627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 10601
I20260618 23:32:27.022658 10673 sys_catalog.cc:458] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b [sys.catalog]: This master's current role is: LEADER
I20260618 23:32:27.022753 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:32:27.022903 10681 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:32:27.022994 392 external_mini_cluster.cc:1428] Setting key 70a0be00e686e1608ba5e732820fef19
I20260618 23:32:27.023463 10681 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:32:27.022308 10674 sys_catalog.cc:455] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b204eb5c1ed64308891e97fb857eb90b. Latest consensus state: current_term: 1 leader_uuid: "b204eb5c1ed64308891e97fb857eb90b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b204eb5c1ed64308891e97fb857eb90b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37961 } } }
I20260618 23:32:27.024227 10674 sys_catalog.cc:458] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b [sys.catalog]: This master's current role is: LEADER
I20260618 23:32:27.025648 10681 catalog_manager.cc:1347] Generated new cluster ID: 28566760e3f84a688b061f116cca2209
I20260618 23:32:27.025717 10681 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 18 23:32:27 dist-test-slave-6224 krb5kdc[10588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:32:27.034703 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:27.023757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41458 (local address 127.0.98.62:37961)
0618 23:32:27.024010 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:27.024023 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:27.024492 (+ 469us) server_negotiation.cc:407] Connection header received
0618 23:32:27.024662 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:27.024671 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:27.024741 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:27.024938 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:27.025825 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.026808 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.027645 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.028171 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.028293 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:27.030899 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:27.030903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:27.030904 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:27.030919 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:27.032849 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:27.033270 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:27.033274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:27.033274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:27.033390 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:27.033613 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:27.033614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:27.033615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:27.033901 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:27.034105 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:27.034193 (+ 88us) server_negotiation.cc:299] Negotiation successful
0618 23:32:27.034481 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":148,"threads_started":1}
I20260618 23:32:27.043218 10681 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:32:27.044117 10681 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:32:27.048688 10681 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b: Generated new TSK 0
I20260618 23:32:27.048929 10681 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260618 23:32:27.082079 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37961
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:32:27.174741 10695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:27.175045 10695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:27.175098 10695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:27.176831 10695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:27.176934 10695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:27.176997 10695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:32:27.178884 10695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:27.179172 10695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:27.179445 10695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:27.184916 10700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:27.185047 10695 server_base.cc:1034] running on GCE node
W20260618 23:32:27.185022 10701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:27.184906 10703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:27.185585 10695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:27.185998 10695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:27.187187 10695 hybrid_clock.cc:648] HybridClock initialized: now 1781825547187164 us; error 51 us; skew 500 ppm
Jun 18 23:32:27 dist-test-slave-6224 krb5kdc[10588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:32:27.190994 10695 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:32:27.191726 10695 webserver.cc:460] Webserver started at http://127.0.98.1:37499/ using document root <none> and password file <none>
I20260618 23:32:27.192037 10695 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:27.192137 10695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:27.192303 10695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:32:27.193611 10695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "9cd6019fa8e74116b5e729029e5180bf"
format_stamp: "Formatted at 2026-06-18 23:32:27 on dist-test-slave-6224"
server_key: "33d9f0983c4aac55d82cd3ed80283976"
server_key_iv: "0ee3fb45ebadd30f4e7595025193fc74"
server_key_version: "encryptionkey@0"
I20260618 23:32:27.194033 10695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9cd6019fa8e74116b5e729029e5180bf"
format_stamp: "Formatted at 2026-06-18 23:32:27 on dist-test-slave-6224"
server_key: "33d9f0983c4aac55d82cd3ed80283976"
server_key_iv: "0ee3fb45ebadd30f4e7595025193fc74"
server_key_version: "encryptionkey@0"
I20260618 23:32:27.195740 10695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260618 23:32:27.196704 10710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:27.196946 10695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:32:27.197072 10695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "9cd6019fa8e74116b5e729029e5180bf"
format_stamp: "Formatted at 2026-06-18 23:32:27 on dist-test-slave-6224"
server_key: "33d9f0983c4aac55d82cd3ed80283976"
server_key_iv: "0ee3fb45ebadd30f4e7595025193fc74"
server_key_version: "encryptionkey@0"
I20260618 23:32:27.197160 10695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:27.216224 10695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:27.218636 10695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:27.218811 10695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:27.219069 10695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:32:27.219412 10695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:32:27.219471 10695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:27.219511 10695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:32:27.219537 10695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:27.226854 10695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37127
I20260618 23:32:27.226984 10823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37127 every 8 connection(s)
I20260618 23:32:27.227324 10695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:32:27.229363 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 10695
I20260618 23:32:27.229498 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:32:27.229748 392 external_mini_cluster.cc:1428] Setting key 19f3dab21660867ff206f9c7aa02135c
Jun 18 23:32:27 dist-test-slave-6224 krb5kdc[10588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260618 23:32:27.240072 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:27.228266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39777 (local address 127.0.98.62:37961)
0618 23:32:27.228378 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:27.228381 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:27.228585 (+ 204us) server_negotiation.cc:407] Connection header received
0618 23:32:27.229267 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:27.229268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:27.229283 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:27.229349 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:27.230881 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.231400 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.232892 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.233235 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.233328 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:27.236312 (+ 2984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:27.236314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:27.236314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:27.236326 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:27.238317 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:27.238916 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:27.238917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:27.238918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:27.238981 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:27.239333 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:27.239334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:27.239335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:27.239494 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:27.239623 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:27.239928 (+ 305us) server_negotiation.cc:299] Negotiation successful
0618 23:32:27.240001 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260618 23:32:27.240443 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:27.228202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37961 (local address 127.0.98.1:39777)
0618 23:32:27.228412 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:27.228465 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:27.229121 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:27.229470 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:27.229477 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:27.230385 (+ 908us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:27.230770 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:27.230778 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.231510 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.231511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:27.232790 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:27.232792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.233368 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.233370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:27.233508 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:27.234187 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:27.234198 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:27.236184 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:27.238421 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:27.238428 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:27.238430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:27.238796 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:27.239086 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:27.239087 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:27.239088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:27.239206 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:27.239613 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:27.239624 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:27.239777 (+ 153us) client_negotiation.cc:769] Sending connection context
0618 23:32:27.239880 (+ 103us) client_negotiation.cc:240] Negotiation successful
0618 23:32:27.240071 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":89,"threads_started":1}
I20260618 23:32:27.241183 10824 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37961
I20260618 23:32:27.241369 10824 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260618 23:32:27.241640 10824 heartbeater.cc:511] Master 127.0.98.62:37961 requested a full tablet report, sending...
I20260618 23:32:27.242280 10633 ts_manager.cc:194] Registered new tserver with Master: 9cd6019fa8e74116b5e729029e5180bf (127.0.98.1:37127)
I20260618 23:32:27.242926 10633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260618 23:32:27.256256 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37961
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:32:27.337608 10829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:27.337849 10829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:27.337885 10829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:27.339241 10829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:27.339304 10829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:27.339350 10829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:32:27.340838 10829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:27.341115 10829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:27.341356 10829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:27.346964 10835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:27.346973 10837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:27.346956 10834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:27.347255 10829 server_base.cc:1034] running on GCE node
I20260618 23:32:27.347666 10829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:27.347980 10829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:27.349179 10829 hybrid_clock.cc:648] HybridClock initialized: now 1781825547349157 us; error 37 us; skew 500 ppm
Jun 18 23:32:27 dist-test-slave-6224 krb5kdc[10588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:32:27.352680 10829 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:32:27.353256 10829 webserver.cc:460] Webserver started at http://127.0.98.2:40607/ using document root <none> and password file <none>
I20260618 23:32:27.353523 10829 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:27.353596 10829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:27.353760 10829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:32:27.354976 10829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "1f859100ae0d4b45a2e57da7270b3557"
format_stamp: "Formatted at 2026-06-18 23:32:27 on dist-test-slave-6224"
server_key: "c83a7ce5b40db09efdc2ddc1dc495692"
server_key_iv: "022bd61198fb034ef021ba0bd810103f"
server_key_version: "encryptionkey@0"
I20260618 23:32:27.355384 10829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1f859100ae0d4b45a2e57da7270b3557"
format_stamp: "Formatted at 2026-06-18 23:32:27 on dist-test-slave-6224"
server_key: "c83a7ce5b40db09efdc2ddc1dc495692"
server_key_iv: "022bd61198fb034ef021ba0bd810103f"
server_key_version: "encryptionkey@0"
I20260618 23:32:27.357059 10829 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:32:27.358147 10844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:27.358462 10829 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:27.358608 10829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "1f859100ae0d4b45a2e57da7270b3557"
format_stamp: "Formatted at 2026-06-18 23:32:27 on dist-test-slave-6224"
server_key: "c83a7ce5b40db09efdc2ddc1dc495692"
server_key_iv: "022bd61198fb034ef021ba0bd810103f"
server_key_version: "encryptionkey@0"
I20260618 23:32:27.358716 10829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:27.379628 10829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:27.382576 10829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:27.382717 10829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:27.382917 10829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:32:27.383311 10829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:32:27.383352 10829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:27.383376 10829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:32:27.383389 10829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:27.390275 10829 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33243
I20260618 23:32:27.390413 10957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33243 every 8 connection(s)
I20260618 23:32:27.390720 10829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:32:27.392588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 10829
I20260618 23:32:27.392714 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:32:27.392969 392 external_mini_cluster.cc:1428] Setting key e21056cf9e279ab4d7e8f7ebf6637cb8
Jun 18 23:32:27 dist-test-slave-6224 krb5kdc[10588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260618 23:32:27.403793 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:27.391709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39061 (local address 127.0.98.62:37961)
0618 23:32:27.391812 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:27.391815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:27.391989 (+ 174us) server_negotiation.cc:407] Connection header received
0618 23:32:27.392810 (+ 821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:27.392812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:27.392829 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:27.392920 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:27.394378 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.394915 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.396392 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.396740 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.396836 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:27.399274 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:27.399276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:27.399276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:27.399296 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:27.401372 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:27.401910 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:27.401912 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:27.401912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:27.401980 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:27.402846 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:27.402847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:27.402848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:27.403015 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:27.403121 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:27.403626 (+ 505us) server_negotiation.cc:299] Negotiation successful
0618 23:32:27.403721 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260618 23:32:27.404040 10960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:27.391629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37961 (local address 127.0.98.2:39061)
0618 23:32:27.391812 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:27.391859 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:27.392627 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:27.393025 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:27.393032 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:27.393900 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:27.394267 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:27.394273 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.395023 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.395024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:27.396293 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:27.396295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.396842 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.396843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:27.396962 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:27.397607 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:27.397616 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:27.399163 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:27.401508 (+ 2345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:27.401516 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:27.401518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:27.401806 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:27.402145 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:27.402146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:27.402147 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:27.402734 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:27.403136 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:27.403146 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:27.403337 (+ 191us) client_negotiation.cc:769] Sending connection context
0618 23:32:27.403492 (+ 155us) client_negotiation.cc:240] Negotiation successful
0618 23:32:27.403736 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260618 23:32:27.404709 10958 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37961
I20260618 23:32:27.404830 10958 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:27.405107 10958 heartbeater.cc:511] Master 127.0.98.62:37961 requested a full tablet report, sending...
I20260618 23:32:27.405727 10633 ts_manager.cc:194] Registered new tserver with Master: 1f859100ae0d4b45a2e57da7270b3557 (127.0.98.2:33243)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260618 23:32:27.406309 10633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39061
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260618 23:32:27.422242 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37961
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:32:27.511865 10963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:32:27.512089 10963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:32:27.512113 10963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:32:27.513288 10963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:32:27.513337 10963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:32:27.513365 10963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:32:27.514833 10963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:32:27.515102 10963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:32:27.515363 10963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:32:27.521129 10968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:32:27.521150 10969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:27.521113 10963 server_base.cc:1034] running on GCE node
W20260618 23:32:27.521119 10971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:32:27.521804 10963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:32:27.522126 10963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:32:27.523310 10963 hybrid_clock.cc:648] HybridClock initialized: now 1781825547523298 us; error 49 us; skew 500 ppm
Jun 18 23:32:27 dist-test-slave-6224 krb5kdc[10588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:32:27.526351 10963 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:32:27.526990 10963 webserver.cc:460] Webserver started at http://127.0.98.3:33215/ using document root <none> and password file <none>
I20260618 23:32:27.527272 10963 fs_manager.cc:359] Metadata directory not provided
I20260618 23:32:27.527346 10963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:32:27.527518 10963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:32:27.528825 10963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "ab30176b1f1a4bcc87b67fcfaf948391"
format_stamp: "Formatted at 2026-06-18 23:32:27 on dist-test-slave-6224"
server_key: "356dbb1a9c39749765e6cd08312f4c2a"
server_key_iv: "f65106000c5e9d4fa73906ba5096c1ec"
server_key_version: "encryptionkey@0"
I20260618 23:32:27.529176 10963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ab30176b1f1a4bcc87b67fcfaf948391"
format_stamp: "Formatted at 2026-06-18 23:32:27 on dist-test-slave-6224"
server_key: "356dbb1a9c39749765e6cd08312f4c2a"
server_key_iv: "f65106000c5e9d4fa73906ba5096c1ec"
server_key_version: "encryptionkey@0"
I20260618 23:32:27.530722 10963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260618 23:32:27.531773 10978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:27.532052 10963 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:32:27.532150 10963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "ab30176b1f1a4bcc87b67fcfaf948391"
format_stamp: "Formatted at 2026-06-18 23:32:27 on dist-test-slave-6224"
server_key: "356dbb1a9c39749765e6cd08312f4c2a"
server_key_iv: "f65106000c5e9d4fa73906ba5096c1ec"
server_key_version: "encryptionkey@0"
I20260618 23:32:27.532232 10963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:32:27.546833 10963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:32:27.549547 10963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:32:27.549686 10963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:32:27.549922 10963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:32:27.550307 10963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:32:27.550348 10963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:27.550374 10963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:32:27.550387 10963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:32:27.558027 10963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33635
I20260618 23:32:27.558156 11091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33635 every 8 connection(s)
I20260618 23:32:27.558538 10963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
Jun 18 23:32:27 dist-test-slave-6224 krb5kdc[10588](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:32:27.568311 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 10963
I20260618 23:32:27.568454 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:32:27.568671 392 external_mini_cluster.cc:1428] Setting key 1f479130b6135ebd4fcce7221b056600
I20260618 23:32:27.569175 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:27.559436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46473 (local address 127.0.98.62:37961)
0618 23:32:27.559509 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:27.559511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:27.559798 (+ 287us) server_negotiation.cc:407] Connection header received
0618 23:32:27.560385 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:27.560386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:27.560396 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:27.560478 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:27.561610 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.562211 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.563307 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.563505 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.563628 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:27.566067 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:27.566068 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:27.566069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:27.566127 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:27.567515 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:27.568077 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:27.568078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:27.568079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:27.568137 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:27.568405 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:27.568406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:27.568407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:27.568569 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:27.568631 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:27.568985 (+ 354us) server_negotiation.cc:299] Negotiation successful
0618 23:32:27.569066 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 23:32:27.569393 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:27.559421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37961 (local address 127.0.98.3:46473)
0618 23:32:27.559636 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:27.559686 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:27.560280 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:27.560592 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:27.560596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:27.561249 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:32:27.561489 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:27.561494 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.562302 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.562303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:27.563214 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:27.563217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.563634 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.563635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:27.563775 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:27.564338 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:32:27.564347 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:32:27.565958 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:32:27.567710 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:27.567718 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:27.567721 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:27.567978 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:27.568210 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:32:27.568211 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:32:27.568211 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:32:27.568328 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:32:27.568637 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:32:27.568643 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:32:27.568837 (+ 194us) client_negotiation.cc:769] Sending connection context
0618 23:32:27.568952 (+ 115us) client_negotiation.cc:240] Negotiation successful
0618 23:32:27.569142 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":136,"threads_started":1}
I20260618 23:32:27.569859 11092 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37961
I20260618 23:32:27.570011 11092 heartbeater.cc:464] Registering TS with master...
I20260618 23:32:27.570222 11092 heartbeater.cc:511] Master 127.0.98.62:37961 requested a full tablet report, sending...
I20260618 23:32:27.570674 10633 ts_manager.cc:194] Registered new tserver with Master: ab30176b1f1a4bcc87b67fcfaf948391 (127.0.98.3:33635)
I20260618 23:32:27.571136 10633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46473
I20260618 23:32:27.581139 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:32:27.589426 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:27.582958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41460 (local address 127.0.98.62:37961)
0618 23:32:27.583051 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:27.583054 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:27.583140 (+ 86us) server_negotiation.cc:407] Connection header received
0618 23:32:27.583248 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:27.583249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:27.583259 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:27.583301 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:32:27.584033 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.584567 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.585308 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.585498 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.585572 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:27.586909 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:32:27.586910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:32:27.586911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:32:27.586918 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:32:27.588215 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:27.588586 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:27.588587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:27.588587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:27.588649 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:32:27.588844 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:32:27.588845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:32:27.588846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:32:27.588999 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:32:27.589111 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:27.589241 (+ 130us) server_negotiation.cc:299] Negotiation successful
0618 23:32:27.589318 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 23:32:27.591406 10633 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:32:27.593437 10633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:32:27.600739 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:27.596370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33635 (local address 127.0.0.1:46908)
0618 23:32:27.597111 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:27.597121 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:27.597171 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:27.597384 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:27.597386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:27.597388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:27.597465 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:27.597467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.598416 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.598417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:27.599532 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:27.599535 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.600376 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.600377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:27.600467 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:27.600485 (+ 18us) client_negotiation.cc:769] Sending connection context
0618 23:32:27.600578 (+ 93us) client_negotiation.cc:240] Negotiation successful
0618 23:32:27.600642 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":97,"threads_started":1}
I20260618 23:32:27.600764 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:27.596600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46908 (local address 127.0.98.3:33635)
0618 23:32:27.596815 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:27.596820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:27.597228 (+ 408us) server_negotiation.cc:407] Connection header received
0618 23:32:27.597282 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:27.597285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:27.597304 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:27.597388 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:27.597533 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.598302 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.599671 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.600270 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.600356 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:27.600424 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:27.600603 (+ 179us) server_negotiation.cc:299] Negotiation successful
0618 23:32:27.600656 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":84,"threads_started":1}
I20260618 23:32:27.601527 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:27.596134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37127 (local address 127.0.0.1:35658)
0618 23:32:27.596646 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:27.596707 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:27.596808 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:27.597245 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:27.597250 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:27.597253 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:27.597334 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:27.597336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.598739 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.598740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:27.600146 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:27.600149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.601319 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.601320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:27.601372 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:27.601376 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:32:27.601421 (+ 45us) client_negotiation.cc:240] Negotiation successful
0618 23:32:27.601457 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":98,"thread_start_us":74,"threads_started":1}
I20260618 23:32:27.601538 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:27.596259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45442 (local address 127.0.98.2:33243)
0618 23:32:27.596479 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:27.596487 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:27.597055 (+ 568us) server_negotiation.cc:407] Connection header received
0618 23:32:27.597122 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:27.597124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:27.597140 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:27.597212 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:27.597444 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.598622 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.600253 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.601087 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.601174 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:27.601233 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:27.601370 (+ 137us) server_negotiation.cc:299] Negotiation successful
0618 23:32:27.601453 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":82,"threads_started":1}
I20260618 23:32:27.601779 11104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:27.596149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33243 (local address 127.0.0.1:45442)
0618 23:32:27.596932 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:27.596940 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:27.597040 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:27.597286 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:27.597287 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:27.597290 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:27.597334 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:27.597336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.598739 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.598740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:27.600147 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:27.600149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.601191 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.601192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:27.601255 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:27.601258 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:32:27.601656 (+ 398us) client_negotiation.cc:240] Negotiation successful
0618 23:32:27.601712 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":43,"thread_start_us":111,"threads_started":1}
I20260618 23:32:27.601841 11026 tablet_service.cc:1467] Processing CreateTablet for tablet 09bed56d88254ddc8cefaa17943167b1 (DEFAULT_TABLE table=test-table [id=59a9df4a9d204e578919c328e5da8253]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:32:27.602281 11026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09bed56d88254ddc8cefaa17943167b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:27.602416 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:27.596289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35658 (local address 127.0.98.1:37127)
0618 23:32:27.596585 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:27.596588 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:27.596798 (+ 210us) server_negotiation.cc:407] Connection header received
0618 23:32:27.596965 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:27.596969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:27.596995 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:27.597122 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:27.597455 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.598618 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.600244 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.601165 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.601230 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:27.601289 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:27.602203 (+ 914us) server_negotiation.cc:299] Negotiation successful
0618 23:32:27.602249 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":161,"threads_started":1}
I20260618 23:32:27.602813 10892 tablet_service.cc:1467] Processing CreateTablet for tablet 09bed56d88254ddc8cefaa17943167b1 (DEFAULT_TABLE table=test-table [id=59a9df4a9d204e578919c328e5da8253]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:32:27.603205 10892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09bed56d88254ddc8cefaa17943167b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:27.603247 10758 tablet_service.cc:1467] Processing CreateTablet for tablet 09bed56d88254ddc8cefaa17943167b1 (DEFAULT_TABLE table=test-table [id=59a9df4a9d204e578919c328e5da8253]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:32:27.603504 10758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09bed56d88254ddc8cefaa17943167b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:32:27.605615 11109 tablet_bootstrap.cc:492] T 09bed56d88254ddc8cefaa17943167b1 P ab30176b1f1a4bcc87b67fcfaf948391: Bootstrap starting.
I20260618 23:32:27.606392 11109 tablet_bootstrap.cc:654] T 09bed56d88254ddc8cefaa17943167b1 P ab30176b1f1a4bcc87b67fcfaf948391: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:27.606956 11110 tablet_bootstrap.cc:492] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf: Bootstrap starting.
I20260618 23:32:27.607220 11109 log.cc:824] T 09bed56d88254ddc8cefaa17943167b1 P ab30176b1f1a4bcc87b67fcfaf948391: Log is configured to *not* fsync() on all Append() calls
I20260618 23:32:27.606725 11111 tablet_bootstrap.cc:492] T 09bed56d88254ddc8cefaa17943167b1 P 1f859100ae0d4b45a2e57da7270b3557: Bootstrap starting.
I20260618 23:32:27.607800 11110 tablet_bootstrap.cc:654] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:27.608310 11109 tablet_bootstrap.cc:492] T 09bed56d88254ddc8cefaa17943167b1 P ab30176b1f1a4bcc87b67fcfaf948391: No bootstrap required, opened a new log
I20260618 23:32:27.608410 11109 ts_tablet_manager.cc:1397] T 09bed56d88254ddc8cefaa17943167b1 P ab30176b1f1a4bcc87b67fcfaf948391: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 23:32:27.608433 11110 log.cc:824] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf: Log is configured to *not* fsync() on all Append() calls
I20260618 23:32:27.608716 11111 tablet_bootstrap.cc:654] T 09bed56d88254ddc8cefaa17943167b1 P 1f859100ae0d4b45a2e57da7270b3557: Neither blocks nor log segments found. Creating new log.
I20260618 23:32:27.609314 11111 log.cc:824] T 09bed56d88254ddc8cefaa17943167b1 P 1f859100ae0d4b45a2e57da7270b3557: Log is configured to *not* fsync() on all Append() calls
I20260618 23:32:27.609390 11110 tablet_bootstrap.cc:492] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf: No bootstrap required, opened a new log
I20260618 23:32:27.609452 11110 ts_tablet_manager.cc:1397] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 23:32:27.610175 11111 tablet_bootstrap.cc:492] T 09bed56d88254ddc8cefaa17943167b1 P 1f859100ae0d4b45a2e57da7270b3557: No bootstrap required, opened a new log
I20260618 23:32:27.610267 11111 ts_tablet_manager.cc:1397] T 09bed56d88254ddc8cefaa17943167b1 P 1f859100ae0d4b45a2e57da7270b3557: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260618 23:32:27.610463 11109 raft_consensus.cc:348] T 09bed56d88254ddc8cefaa17943167b1 P ab30176b1f1a4bcc87b67fcfaf948391 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd6019fa8e74116b5e729029e5180bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37127 } } peers { permanent_uuid: "1f859100ae0d4b45a2e57da7270b3557" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33243 } } peers { permanent_uuid: "ab30176b1f1a4bcc87b67fcfaf948391" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33635 } }
I20260618 23:32:27.610664 11109 raft_consensus.cc:374] T 09bed56d88254ddc8cefaa17943167b1 P ab30176b1f1a4bcc87b67fcfaf948391 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:27.610699 11109 raft_consensus.cc:729] T 09bed56d88254ddc8cefaa17943167b1 P ab30176b1f1a4bcc87b67fcfaf948391 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab30176b1f1a4bcc87b67fcfaf948391, State: Initialized, Role: FOLLOWER
I20260618 23:32:27.610855 11109 consensus_queue.cc:260] T 09bed56d88254ddc8cefaa17943167b1 P ab30176b1f1a4bcc87b67fcfaf948391 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd6019fa8e74116b5e729029e5180bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37127 } } peers { permanent_uuid: "1f859100ae0d4b45a2e57da7270b3557" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33243 } } peers { permanent_uuid: "ab30176b1f1a4bcc87b67fcfaf948391" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33635 } }
I20260618 23:32:27.611177 11109 ts_tablet_manager.cc:1428] T 09bed56d88254ddc8cefaa17943167b1 P ab30176b1f1a4bcc87b67fcfaf948391: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260618 23:32:27.611311 11092 heartbeater.cc:503] Master 127.0.98.62:37961 was elected leader, sending a full tablet report...
I20260618 23:32:27.611506 11110 raft_consensus.cc:348] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd6019fa8e74116b5e729029e5180bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37127 } } peers { permanent_uuid: "1f859100ae0d4b45a2e57da7270b3557" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33243 } } peers { permanent_uuid: "ab30176b1f1a4bcc87b67fcfaf948391" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33635 } }
I20260618 23:32:27.611744 11110 raft_consensus.cc:374] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:27.611791 11110 raft_consensus.cc:729] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cd6019fa8e74116b5e729029e5180bf, State: Initialized, Role: FOLLOWER
I20260618 23:32:27.611927 11110 consensus_queue.cc:260] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd6019fa8e74116b5e729029e5180bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37127 } } peers { permanent_uuid: "1f859100ae0d4b45a2e57da7270b3557" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33243 } } peers { permanent_uuid: "ab30176b1f1a4bcc87b67fcfaf948391" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33635 } }
I20260618 23:32:27.612272 11110 ts_tablet_manager.cc:1428] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:32:27.612402 10824 heartbeater.cc:503] Master 127.0.98.62:37961 was elected leader, sending a full tablet report...
I20260618 23:32:27.612371 11111 raft_consensus.cc:348] T 09bed56d88254ddc8cefaa17943167b1 P 1f859100ae0d4b45a2e57da7270b3557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd6019fa8e74116b5e729029e5180bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37127 } } peers { permanent_uuid: "1f859100ae0d4b45a2e57da7270b3557" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33243 } } peers { permanent_uuid: "ab30176b1f1a4bcc87b67fcfaf948391" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33635 } }
I20260618 23:32:27.612531 11111 raft_consensus.cc:374] T 09bed56d88254ddc8cefaa17943167b1 P 1f859100ae0d4b45a2e57da7270b3557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:32:27.612577 11111 raft_consensus.cc:729] T 09bed56d88254ddc8cefaa17943167b1 P 1f859100ae0d4b45a2e57da7270b3557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f859100ae0d4b45a2e57da7270b3557, State: Initialized, Role: FOLLOWER
I20260618 23:32:27.612679 11111 consensus_queue.cc:260] T 09bed56d88254ddc8cefaa17943167b1 P 1f859100ae0d4b45a2e57da7270b3557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd6019fa8e74116b5e729029e5180bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37127 } } peers { permanent_uuid: "1f859100ae0d4b45a2e57da7270b3557" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33243 } } peers { permanent_uuid: "ab30176b1f1a4bcc87b67fcfaf948391" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33635 } }
I20260618 23:32:27.612986 11111 ts_tablet_manager.cc:1428] T 09bed56d88254ddc8cefaa17943167b1 P 1f859100ae0d4b45a2e57da7270b3557: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:32:27.613133 10958 heartbeater.cc:503] Master 127.0.98.62:37961 was elected leader, sending a full tablet report...
W20260618 23:32:27.641382 10959 tablet.cc:2369] T 09bed56d88254ddc8cefaa17943167b1 P 1f859100ae0d4b45a2e57da7270b3557: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:32:27.728210 10825 tablet.cc:2369] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:32:27.809199 11093 tablet.cc:2369] T 09bed56d88254ddc8cefaa17943167b1 P ab30176b1f1a4bcc87b67fcfaf948391: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:32:27.925545 11116 raft_consensus.cc:482] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:32:27.925685 11116 raft_consensus.cc:504] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd6019fa8e74116b5e729029e5180bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37127 } } peers { permanent_uuid: "1f859100ae0d4b45a2e57da7270b3557" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33243 } } peers { permanent_uuid: "ab30176b1f1a4bcc87b67fcfaf948391" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33635 } }
I20260618 23:32:27.926013 11116 leader_election.cc:288] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f859100ae0d4b45a2e57da7270b3557 (127.0.98.2:33243), ab30176b1f1a4bcc87b67fcfaf948391 (127.0.98.3:33635)
I20260618 23:32:27.929644 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:27.926419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51825 (local address 127.0.98.2:33243)
0618 23:32:27.926509 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:27.926512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:27.926652 (+ 140us) server_negotiation.cc:407] Connection header received
0618 23:32:27.926788 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:27.926790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:27.926806 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:27.926872 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:27.927087 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.927692 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.928821 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.929228 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.929290 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:27.929302 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:27.929502 (+ 200us) server_negotiation.cc:299] Negotiation successful
0618 23:32:27.929554 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 23:32:27.929656 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:27.926288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33243 (local address 127.0.98.1:51825)
0618 23:32:27.926505 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:27.926521 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:27.926678 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:27.926899 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:27.926900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:27.926903 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:27.926996 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:27.926999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.927785 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.927786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:27.928719 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:27.928721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.929329 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.929330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:27.929408 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:27.929414 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:32:27.929476 (+ 62us) client_negotiation.cc:240] Negotiation successful
0618 23:32:27.929528 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":120,"threads_started":1}
I20260618 23:32:27.929728 11108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:27.926736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46009 (local address 127.0.98.3:33635)
0618 23:32:27.926796 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:27.926799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:27.926808 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:32:27.926865 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:27.926867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:27.926877 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:27.926942 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:32:27.927199 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.927829 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.928897 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.929303 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.929379 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:27.929391 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:27.929593 (+ 202us) server_negotiation.cc:299] Negotiation successful
0618 23:32:27.929642 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 23:32:27.929812 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:27.926556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33635 (local address 127.0.98.1:46009)
0618 23:32:27.926663 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:27.926684 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 23:32:27.926777 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:32:27.926965 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:32:27.926966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:32:27.926969 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:32:27.927109 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:27.927112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.927920 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.927921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:27.928759 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:32:27.928785 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.929401 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.929401 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:32:27.929483 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:27.929489 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:32:27.929564 (+ 75us) client_negotiation.cc:240] Negotiation successful
0618 23:32:27.929618 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":16,"thread_start_us":58,"threads_started":1}
I20260618 23:32:27.930150 11046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09bed56d88254ddc8cefaa17943167b1" candidate_uuid: "9cd6019fa8e74116b5e729029e5180bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab30176b1f1a4bcc87b67fcfaf948391" is_pre_election: true
I20260618 23:32:27.930161 10912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09bed56d88254ddc8cefaa17943167b1" candidate_uuid: "9cd6019fa8e74116b5e729029e5180bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f859100ae0d4b45a2e57da7270b3557" is_pre_election: true
I20260618 23:32:27.930330 11046 raft_consensus.cc:2454] T 09bed56d88254ddc8cefaa17943167b1 P ab30176b1f1a4bcc87b67fcfaf948391 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cd6019fa8e74116b5e729029e5180bf in term 0.
I20260618 23:32:27.930331 10912 raft_consensus.cc:2454] T 09bed56d88254ddc8cefaa17943167b1 P 1f859100ae0d4b45a2e57da7270b3557 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cd6019fa8e74116b5e729029e5180bf in term 0.
I20260618 23:32:27.930598 10714 leader_election.cc:302] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f859100ae0d4b45a2e57da7270b3557, 9cd6019fa8e74116b5e729029e5180bf; no voters:
I20260618 23:32:27.930742 11116 raft_consensus.cc:2781] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:32:27.930799 11116 raft_consensus.cc:482] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:32:27.930825 11116 raft_consensus.cc:3037] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:27.931573 11116 raft_consensus.cc:504] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd6019fa8e74116b5e729029e5180bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37127 } } peers { permanent_uuid: "1f859100ae0d4b45a2e57da7270b3557" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33243 } } peers { permanent_uuid: "ab30176b1f1a4bcc87b67fcfaf948391" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33635 } }
I20260618 23:32:27.931731 11116 leader_election.cc:288] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf [CANDIDATE]: Term 1 election: Requested vote from peers 1f859100ae0d4b45a2e57da7270b3557 (127.0.98.2:33243), ab30176b1f1a4bcc87b67fcfaf948391 (127.0.98.3:33635)
I20260618 23:32:27.931917 10912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09bed56d88254ddc8cefaa17943167b1" candidate_uuid: "9cd6019fa8e74116b5e729029e5180bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f859100ae0d4b45a2e57da7270b3557"
I20260618 23:32:27.931948 11046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09bed56d88254ddc8cefaa17943167b1" candidate_uuid: "9cd6019fa8e74116b5e729029e5180bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab30176b1f1a4bcc87b67fcfaf948391"
I20260618 23:32:27.932057 11046 raft_consensus.cc:3037] T 09bed56d88254ddc8cefaa17943167b1 P ab30176b1f1a4bcc87b67fcfaf948391 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:27.932057 10912 raft_consensus.cc:3037] T 09bed56d88254ddc8cefaa17943167b1 P 1f859100ae0d4b45a2e57da7270b3557 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:32:27.933215 11046 raft_consensus.cc:2454] T 09bed56d88254ddc8cefaa17943167b1 P ab30176b1f1a4bcc87b67fcfaf948391 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cd6019fa8e74116b5e729029e5180bf in term 1.
I20260618 23:32:27.933215 10912 raft_consensus.cc:2454] T 09bed56d88254ddc8cefaa17943167b1 P 1f859100ae0d4b45a2e57da7270b3557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cd6019fa8e74116b5e729029e5180bf in term 1.
I20260618 23:32:27.933416 10714 leader_election.cc:302] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f859100ae0d4b45a2e57da7270b3557, 9cd6019fa8e74116b5e729029e5180bf; no voters:
I20260618 23:32:27.933552 11116 raft_consensus.cc:2781] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:32:27.933765 11116 raft_consensus.cc:686] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf [term 1 LEADER]: Becoming Leader. State: Replica: 9cd6019fa8e74116b5e729029e5180bf, State: Running, Role: LEADER
I20260618 23:32:27.933884 11116 consensus_queue.cc:237] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd6019fa8e74116b5e729029e5180bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37127 } } peers { permanent_uuid: "1f859100ae0d4b45a2e57da7270b3557" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33243 } } peers { permanent_uuid: "ab30176b1f1a4bcc87b67fcfaf948391" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33635 } }
I20260618 23:32:27.934908 10632 catalog_manager.cc:5495] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cd6019fa8e74116b5e729029e5180bf (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9cd6019fa8e74116b5e729029e5180bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cd6019fa8e74116b5e729029e5180bf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f859100ae0d4b45a2e57da7270b3557" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab30176b1f1a4bcc87b67fcfaf948391" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33635 } health_report { overall_health: UNKNOWN } } }
I20260618 23:32:27.957206 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:27.954609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35660 (local address 127.0.98.1:37127)
0618 23:32:27.954694 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0618 23:32:27.954696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:32:27.954705 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:32:27.954758 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:32:27.954759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:32:27.954785 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:32:27.954861 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:32:27.955010 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.955518 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.956391 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:32:27.956585 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:32:27.956645 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:32:27.956798 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:32:27.956926 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:32:27.956998 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 23:32:27.957052 (+ 54us) server_negotiation.cc:299] Negotiation successful
0618 23:32:27.957074 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 23:32:27.959645 11046 raft_consensus.cc:1261] T 09bed56d88254ddc8cefaa17943167b1 P ab30176b1f1a4bcc87b67fcfaf948391 [term 1 FOLLOWER]: Refusing update from remote peer 9cd6019fa8e74116b5e729029e5180bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:32:27.959707 10912 raft_consensus.cc:1261] T 09bed56d88254ddc8cefaa17943167b1 P 1f859100ae0d4b45a2e57da7270b3557 [term 1 FOLLOWER]: Refusing update from remote peer 9cd6019fa8e74116b5e729029e5180bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:32:27.960103 11116 consensus_queue.cc:1035] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab30176b1f1a4bcc87b67fcfaf948391" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:32:27.960295 11120 consensus_queue.cc:1035] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f859100ae0d4b45a2e57da7270b3557" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:32:27.962841 11123 mvcc.cc:205] Tried to move back new op lower bound from 7298357444439138304 to 7298357444337811456. Current Snapshot: MvccSnapshot[applied={T|T < 7298357444439138304}]
I20260618 23:32:27.963779 11125 mvcc.cc:205] Tried to move back new op lower bound from 7298357444439138304 to 7298357444337811456. Current Snapshot: MvccSnapshot[applied={T|T < 7298357444439138304}]
I20260618 23:32:27.964062 11124 mvcc.cc:205] Tried to move back new op lower bound from 7298357444439138304 to 7298357444337811456. Current Snapshot: MvccSnapshot[applied={T|T < 7298357444439138304}]
I20260618 23:32:27.965426 10633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:32:27.965554 10633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:32:27.966588 10633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b204eb5c1ed64308891e97fb857eb90b: Sending DeleteTablet for 3 replicas of tablet 09bed56d88254ddc8cefaa17943167b1
I20260618 23:32:27.967103 11026 tablet_service.cc:1514] Processing DeleteTablet for tablet 09bed56d88254ddc8cefaa17943167b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:32:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46908
I20260618 23:32:27.967128 10758 tablet_service.cc:1514] Processing DeleteTablet for tablet 09bed56d88254ddc8cefaa17943167b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:32:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35658
I20260618 23:32:27.967355 10892 tablet_service.cc:1514] Processing DeleteTablet for tablet 09bed56d88254ddc8cefaa17943167b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:32:27 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45442
I20260618 23:32:27.967396 11133 tablet_replica.cc:330] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf: stopping tablet replica
I20260618 23:32:27.967509 11133 raft_consensus.cc:2229] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:32:27.967584 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 10695
I20260618 23:32:27.967649 11133 raft_consensus.cc:2258] T 09bed56d88254ddc8cefaa17943167b1 P 9cd6019fa8e74116b5e729029e5180bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:32:27.968384 11134 tablet_replica.cc:330] T 09bed56d88254ddc8cefaa17943167b1 P 1f859100ae0d4b45a2e57da7270b3557: stopping tablet replica
I20260618 23:32:27.968493 11134 raft_consensus.cc:2229] T 09bed56d88254ddc8cefaa17943167b1 P 1f859100ae0d4b45a2e57da7270b3557 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:32:27.968606 11134 raft_consensus.cc:2258] T 09bed56d88254ddc8cefaa17943167b1 P 1f859100ae0d4b45a2e57da7270b3557 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:32:27.969095 11134 ts_tablet_manager.cc:1905] T 09bed56d88254ddc8cefaa17943167b1 P 1f859100ae0d4b45a2e57da7270b3557: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:32:27.969276 11132 tablet_replica.cc:330] T 09bed56d88254ddc8cefaa17943167b1 P ab30176b1f1a4bcc87b67fcfaf948391: stopping tablet replica
I20260618 23:32:27.969446 11132 raft_consensus.cc:2229] T 09bed56d88254ddc8cefaa17943167b1 P ab30176b1f1a4bcc87b67fcfaf948391 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:32:27.969614 11132 raft_consensus.cc:2258] T 09bed56d88254ddc8cefaa17943167b1 P ab30176b1f1a4bcc87b67fcfaf948391 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:32:27.970150 11132 ts_tablet_manager.cc:1905] T 09bed56d88254ddc8cefaa17943167b1 P ab30176b1f1a4bcc87b67fcfaf948391: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:32:27.971418 11134 ts_tablet_manager.cc:1918] T 09bed56d88254ddc8cefaa17943167b1 P 1f859100ae0d4b45a2e57da7270b3557: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:32:27.971504 11134 log.cc:1196] T 09bed56d88254ddc8cefaa17943167b1 P 1f859100ae0d4b45a2e57da7270b3557: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/09bed56d88254ddc8cefaa17943167b1
I20260618 23:32:27.971793 11134 ts_tablet_manager.cc:1939] T 09bed56d88254ddc8cefaa17943167b1 P 1f859100ae0d4b45a2e57da7270b3557: Deleting consensus metadata
I20260618 23:32:27.971935 11132 ts_tablet_manager.cc:1918] T 09bed56d88254ddc8cefaa17943167b1 P ab30176b1f1a4bcc87b67fcfaf948391: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:32:27.972014 11132 log.cc:1196] T 09bed56d88254ddc8cefaa17943167b1 P ab30176b1f1a4bcc87b67fcfaf948391: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/09bed56d88254ddc8cefaa17943167b1
I20260618 23:32:27.972323 11132 ts_tablet_manager.cc:1939] T 09bed56d88254ddc8cefaa17943167b1 P ab30176b1f1a4bcc87b67fcfaf948391: Deleting consensus metadata
I20260618 23:32:27.972339 10620 catalog_manager.cc:4841] TS 1f859100ae0d4b45a2e57da7270b3557 (127.0.98.2:33243): tablet 09bed56d88254ddc8cefaa17943167b1 (table test-table [id=59a9df4a9d204e578919c328e5da8253]) successfully deleted
I20260618 23:32:27.972818 10620 catalog_manager.cc:4841] TS ab30176b1f1a4bcc87b67fcfaf948391 (127.0.98.3:33635): tablet 09bed56d88254ddc8cefaa17943167b1 (table test-table [id=59a9df4a9d204e578919c328e5da8253]) successfully deleted
W20260618 23:32:27.973842 10618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37127 (error 108)
I20260618 23:32:27.974098 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 10829
I20260618 23:32:27.974339 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:32:27.974218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37127 (local address 127.0.0.1:35666)
0618 23:32:27.974268 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0618 23:32:27.974300 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260618 23:32:27.974452 10618 catalog_manager.cc:4584] TS 9cd6019fa8e74116b5e729029e5180bf (127.0.98.1:37127): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09bed56d88254ddc8cefaa17943167b1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37127: connect: Connection refused (error 111)
I20260618 23:32:27.979736 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 10963
I20260618 23:32:27.984951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 10601
2026-06-18T23:32:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2270 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:32:28 dist-test-slave-6224 krb5kdc[11138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:32:28 dist-test-slave-6224 krb5kdc[11138](info): set up 2 sockets
Jun 18 23:32:28 dist-test-slave-6224 krb5kdc[11138](info): commencing operation
krb5kdc: starting...
W20260618 23:32:29.064874 392 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.001s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:32:29 dist-test-slave-6224 krb5kdc[11138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:32:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:32:29Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260618 23:32:29.247071 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcnfpmz/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/postgres -l logfile start
2026-06-18 23:32:31.971 UTC [11168] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-18 23:32:31.971 UTC [11168] LOG: listening on IPv4 address "127.0.98.20", port 45967
2026-06-18 23:32:31.975 UTC [11168] LOG: listening on Unix socket "/tmp/.s.PGSQL.45967"
2026-06-18 23:32:32.000 UTC [11171] LOG: database system was shut down at 2026-06-18 23:32:30 UTC
2026-06-18 23:32:32.004 UTC [11168] LOG: database system is ready to accept connections
I20260618 23:32:32.960824 392 mini_postgres.cc:96] Postgres bound to 45967
2026-06-18 23:32:32.967 UTC [11179] FATAL: database "slave" does not exist
127.0.98.20:45967 - accepting connections
I20260618 23:32:32.968430 392 mini_ranger.cc:161] Starting Ranger...
I20260618 23:32:32.985193 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260618 23:32:33.549830 392 mini_ranger.cc:87] Created ranger Postgres database
I20260618 23:32:33.549925 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ranger-admin
2026-06-18 23:32:33,864 [I] DB FLAVOR :POSTGRES
2026-06-18 23:32:33,865 [I] --------- Verifying Ranger DB connection ---------
2026-06-18 23:32:33,865 [I] Checking connection..
2026-06-18 23:32:33,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-18 23:32:34,142 [I] Checking connection passed.
2026-06-18 23:32:34,142 [I] --------- Verifying version history table ---------
2026-06-18 23:32:34,143 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 23:32:34,432 [I] Table x_db_version_h does not exist in database ranger
2026-06-18 23:32:34,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 23:32:34,704 [I] Table x_db_version_h does not exist in database ranger
2026-06-18 23:32:34,704 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-18 23:32:34,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-18 23:32:34.956 UTC [11243] WARNING: there is no transaction in progress
2026-06-18 23:32:34,963 [I] create_dbversion_catalog.sql file imported successfully
2026-06-18 23:32:34,964 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 23:32:35,219 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-18 23:32:35,220 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-18 23:32:35,220 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-18 23:32:35,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-18 23:32:35,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6224.c.gcp-upstream.internal','N') ;"
2026-06-18 23:32:36,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-18 23:32:36,311 [I] Table x_portal_user does not exist in database ranger
2026-06-18 23:32:36,312 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-18 23:32:36,575 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-18 23:32:36,575 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-18 23:32:36,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-18 23:32:37,129 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-18 23:32:37,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-18 23:32:38.208 UTC [11351] WARNING: there is no transaction in progress
2026-06-18 23:32:38.230 UTC [11351] WARNING: there is no transaction in progress
2026-06-18 23:32:38.249 UTC [11351] WARNING: there is no transaction in progress
2026-06-18 23:32:38.267 UTC [11351] WARNING: there is no transaction in progress
2026-06-18 23:32:38.287 UTC [11351] WARNING: there is no transaction in progress
2026-06-18 23:32:38.307 UTC [11351] WARNING: there is no transaction in progress
2026-06-18 23:32:38.373 UTC [11351] WARNING: there is no transaction in progress
2026-06-18 23:32:38.381 UTC [11351] WARNING: there is no transaction in progress
2026-06-18 23:32:38.396 UTC [11351] WARNING: there is no transaction in progress
2026-06-18 23:32:38.407 UTC [11351] WARNING: there is no transaction in progress
2026-06-18 23:32:38.421 UTC [11351] WARNING: there is no transaction in progress
2026-06-18 23:32:38.439 UTC [11351] WARNING: there is no transaction in progress
2026-06-18 23:32:38.452 UTC [11351] WARNING: there is no transaction in progress
2026-06-18 23:32:38.459 UTC [11351] WARNING: there is no transaction in progress
2026-06-18 23:32:38.467 UTC [11351] WARNING: there is no transaction in progress
2026-06-18 23:32:38.474 UTC [11351] WARNING: there is no transaction in progress
2026-06-18 23:32:39,230 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-18 23:32:39,230 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-18 23:32:39,512 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-18 23:32:39,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-18 23:32:39,790 [I] Table x_portal_user already exists in database 'ranger'
2026-06-18 23:32:39,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-18 23:32:40,064 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-18 23:32:40,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-18 23:32:40,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-18 23:32:40,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6224.c.gcp-upstream.internal';"
2026-06-18 23:32:40,855 [I] CORE_DB_SCHEMA import status has been updated
2026-06-18 23:32:40,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-18 23:32:41,125 [I] DB_PATCHES have already been applied
I20260618 23:32:41.131326 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcnfpmz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260618 23:32:41.131434 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260618 23:32:41.135264 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:56607
Jun 18, 2026 11:32:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 18, 2026 11:32:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 18, 2026 11:32:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 18, 2026 11:32:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 18, 2026 11:32:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 18, 2026 11:32:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 18, 2026 11:32:42 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260618 23:32:42.301458 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.333s user 0.000s sys 0.011s
Jun 18, 2026 11:33:03 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 18, 2026 11:33:03 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 18, 2026 11:33:03 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 18, 2026 11:33:03 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 18, 2026 11:33:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 18, 2026 11:33:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 18, 2026 11:33:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 18, 2026 11:33:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 18, 2026 11:33:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 18, 2026 11:33:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 18, 2026 11:33:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 18, 2026 11:33:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 18, 2026 11:33:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 18, 2026 11:33:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 18, 2026 11:33:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 18, 2026 11:33:03 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 18, 2026 11:33:03 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 18, 2026 11:33:04 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260618 23:33:04.873041 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 18 23:33:04 dist-test-slave-6224 krb5kdc[11138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825584, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260618 23:33:04.966130 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260618 23:33:04.976488 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260618 23:33:05.772015 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260618 23:33:05.772147 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ranger-kms
I20260618 23:33:05.772168 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:45967
2026-06-18 23:33:05,986 [I] DB FLAVOR :POSTGRES
2026-06-18 23:33:05,986 [I] --------- Verifying Ranger DB connection ---------
2026-06-18 23:33:05,987 [I] Checking connection
2026-06-18 23:33:05,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-18 23:33:06,260 [I] connection success
2026-06-18 23:33:06,261 [I] --------- Verifying Ranger DB tables ---------
2026-06-18 23:33:06,261 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-18 23:33:06,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-18 23:33:06,520 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-18 23:33:06,521 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-18 23:33:06,521 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-18 23:33:06,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:45967/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-18 23:33:06,783 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260618 23:33:07.244833 392 mini_ranger_kms.cc:310] Created kms service
I20260618 23:33:07.444499 392 mini_ranger_kms.cc:326] Created kudu user
I20260618 23:33:07.487623 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 18 23:33:07 dist-test-slave-6224 krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825584, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260618 23:33:08.006321 392 mini_ranger_kms.cc:384] Added ranger policy
I20260618 23:33:08.006476 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260618 23:33:08.006500 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260618 23:33:08.011427 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11532
I20260618 23:33:08.011651 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:46251
Jun 18, 2026 11:33:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 18, 2026 11:33:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 18, 2026 11:33:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 18, 2026 11:33:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 18, 2026 11:33:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 18, 2026 11:33:09 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260618 23:33:10.332237 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.366s user 0.004s sys 0.196s
I20260618 23:33:10.332365 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260618 23:33:10.332443 392 mini_ranger_kms.cc:401] 127.0.98.20:46251/kms/v1/keys
Jun 18 23:33:10 dist-test-slave-6224 krb5kdc[11138](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781825590, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 18 23:33:10 dist-test-slave-6224 krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825590, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 18 23:33:11 dist-test-slave-6224 krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825590, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 18 23:33:13 dist-test-slave-6224 krb5kdc[11138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:33:13.830538 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40635
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40635
--ranger_config_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46251/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:33:13.924079 11582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:33:13.924321 11582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:33:13.924346 11582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:33:13.925671 11582 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:33:13.925731 11582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:33:13.925746 11582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:33:13.925760 11582 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:33:13.925771 11582 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:33:13.927455 11582 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:46251/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40635
--ranger_config_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:33:13.927788 11582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:33:13.928056 11582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:33:13.934738 11582 server_base.cc:1034] running on GCE node
W20260618 23:33:13.934703 11590 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:33:13.934695 11587 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:33:13.934679 11588 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:33:13.935617 11582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:33:13.936043 11582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:33:13.937211 11582 hybrid_clock.cc:648] HybridClock initialized: now 1781825593937171 us; error 64 us; skew 500 ppm
Jun 18 23:33:13 dist-test-slave-6224 krb5kdc[11138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:33:13.945758 11582 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:33:13.946332 11582 webserver.cc:460] Webserver started at http://127.0.98.62:45339/ using document root <none> and password file <none>
I20260618 23:33:13.946615 11582 fs_manager.cc:359] Metadata directory not provided
I20260618 23:33:13.946681 11582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:33:13.946866 11582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 23:33:13 dist-test-slave-6224 krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260618 23:33:14.017838 11582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "034bbef8728f487aa169891f03e3c2fe"
format_stamp: "Formatted at 2026-06-18 23:33:14 on dist-test-slave-6224"
server_key: "2069ef55f1b085a3795056601a42d88e"
server_key_iv: "8b93f6086336334891429b2d397f8633"
server_key_version: "kuduclusterkey@0"
I20260618 23:33:14.018353 11582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "034bbef8728f487aa169891f03e3c2fe"
format_stamp: "Formatted at 2026-06-18 23:33:14 on dist-test-slave-6224"
server_key: "2069ef55f1b085a3795056601a42d88e"
server_key_iv: "8b93f6086336334891429b2d397f8633"
server_key_version: "kuduclusterkey@0"
I20260618 23:33:14.020463 11582 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260618 23:33:14.036477 11597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:33:14.036795 11582 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260618 23:33:14.037004 11582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "034bbef8728f487aa169891f03e3c2fe"
format_stamp: "Formatted at 2026-06-18 23:33:14 on dist-test-slave-6224"
server_key: "2069ef55f1b085a3795056601a42d88e"
server_key_iv: "8b93f6086336334891429b2d397f8633"
server_key_version: "kuduclusterkey@0"
I20260618 23:33:14.037302 11582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:33:14.060062 11582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:33:14.062573 11582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:33:14.062704 11582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:33:14.067943 11582 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40635
I20260618 23:33:14.067991 11649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40635 every 8 connection(s)
I20260618 23:33:14.068395 11582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:33:14.069337 11650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:33:14.070119 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 11582
I20260618 23:33:14.070268 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:33:14.070458 392 external_mini_cluster.cc:1428] Setting key 0a43c57fdb9aaf89537a7c4a3068f2a4
I20260618 23:33:14.073166 11650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe: Bootstrap starting.
I20260618 23:33:14.074029 11650 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe: Neither blocks nor log segments found. Creating new log.
I20260618 23:33:14.074370 11650 log.cc:824] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe: Log is configured to *not* fsync() on all Append() calls
I20260618 23:33:14.075493 11650 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe: No bootstrap required, opened a new log
I20260618 23:33:14.078698 11650 raft_consensus.cc:348] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034bbef8728f487aa169891f03e3c2fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40635 } }
I20260618 23:33:14.078904 11650 raft_consensus.cc:374] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:33:14.078930 11650 raft_consensus.cc:729] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 034bbef8728f487aa169891f03e3c2fe, State: Initialized, Role: FOLLOWER
Jun 18 23:33:14 dist-test-slave-6224 krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:33:14.079140 11650 consensus_queue.cc:260] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034bbef8728f487aa169891f03e3c2fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40635 } }
I20260618 23:33:14.079234 11650 raft_consensus.cc:388] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:33:14.079267 11650 raft_consensus.cc:482] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:33:14.079316 11650 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:33:14.080184 11650 raft_consensus.cc:504] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034bbef8728f487aa169891f03e3c2fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40635 } }
I20260618 23:33:14.080336 11650 leader_election.cc:302] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 034bbef8728f487aa169891f03e3c2fe; no voters:
I20260618 23:33:14.080595 11650 leader_election.cc:288] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:33:14.080696 11655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:33:14.080928 11650 sys_catalog.cc:564] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:33:14.081295 11655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe [term 1 LEADER]: Becoming Leader. State: Replica: 034bbef8728f487aa169891f03e3c2fe, State: Running, Role: LEADER
I20260618 23:33:14.081458 11655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034bbef8728f487aa169891f03e3c2fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40635 } }
I20260618 23:33:14.082247 11656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "034bbef8728f487aa169891f03e3c2fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034bbef8728f487aa169891f03e3c2fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40635 } } }
I20260618 23:33:14.082348 11656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe [sys.catalog]: This master's current role is: LEADER
I20260618 23:33:14.082587 11657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 034bbef8728f487aa169891f03e3c2fe. Latest consensus state: current_term: 1 leader_uuid: "034bbef8728f487aa169891f03e3c2fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "034bbef8728f487aa169891f03e3c2fe" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40635 } } }
I20260618 23:33:14.082644 11657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe [sys.catalog]: This master's current role is: LEADER
I20260618 23:33:14.082666 11650 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260618 23:33:14.083384 11669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:33:14.087787 11669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:33:14.088475 11653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:33:14.071122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39870 (local address 127.0.98.62:40635)
0618 23:33:14.071374 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0618 23:33:14.071383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:33:14.071417 (+ 34us) server_negotiation.cc:407] Connection header received
0618 23:33:14.072235 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:33:14.072243 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:33:14.072302 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:33:14.072528 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:33:14.073279 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:33:14.074425 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:33:14.075660 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:33:14.076144 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:33:14.076246 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:33:14.079724 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:33:14.079737 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:33:14.079737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:33:14.079751 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:33:14.082175 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:33:14.083105 (+ 930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:33:14.083109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:33:14.083109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:33:14.083210 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:33:14.083592 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:33:14.083593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:33:14.083593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:33:14.086813 (+ 3220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:33:14.087287 (+ 474us) server_negotiation.cc:1035] Waiting for connection context
0618 23:33:14.087432 (+ 145us) server_negotiation.cc:299] Negotiation successful
0618 23:33:14.088048 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":120,"threads_started":1}
I20260618 23:33:14.091636 11669 catalog_manager.cc:1347] Generated new cluster ID: a874af7a6b2c4aa99f242c730ba10509
I20260618 23:33:14.091809 11669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:33:14.105574 11669 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:33:14.106261 11669 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:33:14.110051 11669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe: Generated new TSK 0
I20260618 23:33:14.110268 11669 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260618 23:33:14.931205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46251/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40635
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:33:15.040994 11695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:33:15.041384 11695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:33:15.041426 11695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:33:15.043224 11695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:33:15.043286 11695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:33:15.043321 11695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:33:15.045293 11695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:46251/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:33:15.045593 11695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:33:15.045992 11695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:33:15.054191 11701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:33:15.054363 11703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:33:15.054191 11700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:33:15.054368 11695 server_base.cc:1034] running on GCE node
I20260618 23:33:15.054876 11695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:33:15.055358 11695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:33:15.056540 11695 hybrid_clock.cc:648] HybridClock initialized: now 1781825595056487 us; error 74 us; skew 500 ppm
Jun 18 23:33:15 dist-test-slave-6224 krb5kdc[11138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:33:15.060256 11695 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:33:15.061231 11695 webserver.cc:460] Webserver started at http://127.0.98.1:36587/ using document root <none> and password file <none>
I20260618 23:33:15.061559 11695 fs_manager.cc:359] Metadata directory not provided
I20260618 23:33:15.061617 11695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:33:15.061779 11695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 23:33:15 dist-test-slave-6224 krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260618 23:33:15.081854 11695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "934655bb12d64e46a999167c40d27b70"
format_stamp: "Formatted at 2026-06-18 23:33:15 on dist-test-slave-6224"
server_key: "33c83f24beac16e04a54502c7b75613e"
server_key_iv: "56645a3084c570cac4d4c4ba03896e26"
server_key_version: "kuduclusterkey@0"
I20260618 23:33:15.082434 11695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "934655bb12d64e46a999167c40d27b70"
format_stamp: "Formatted at 2026-06-18 23:33:15 on dist-test-slave-6224"
server_key: "33c83f24beac16e04a54502c7b75613e"
server_key_iv: "56645a3084c570cac4d4c4ba03896e26"
server_key_version: "kuduclusterkey@0"
I20260618 23:33:15.090458 11695 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.000s
I20260618 23:33:15.109634 11712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:33:15.110034 11695 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:33:15.110157 11695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "934655bb12d64e46a999167c40d27b70"
format_stamp: "Formatted at 2026-06-18 23:33:15 on dist-test-slave-6224"
server_key: "33c83f24beac16e04a54502c7b75613e"
server_key_iv: "56645a3084c570cac4d4c4ba03896e26"
server_key_version: "kuduclusterkey@0"
I20260618 23:33:15.110286 11695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:33:15.133002 11695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:33:15.135625 11695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:33:15.135851 11695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:33:15.136159 11695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:33:15.136618 11695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:33:15.136674 11695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:33:15.136754 11695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:33:15.136796 11695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:33:15.146853 11695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43059
I20260618 23:33:15.146934 11825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43059 every 8 connection(s)
I20260618 23:33:15.147400 11695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:33:15.148952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 11695
I20260618 23:33:15.149093 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:33:15.149289 392 external_mini_cluster.cc:1428] Setting key 19e2150e94863cca607e7a06515f4b14
Jun 18 23:33:15 dist-test-slave-6224 krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:33:15.159461 11828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:33:15.148442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58531 (local address 127.0.98.62:40635)
0618 23:33:15.148592 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0618 23:33:15.148594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:33:15.148843 (+ 249us) server_negotiation.cc:407] Connection header received
0618 23:33:15.149726 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:33:15.149727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:33:15.149755 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:33:15.149827 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:33:15.150721 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.151600 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.152975 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.153202 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.153304 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:33:15.155956 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:33:15.155958 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:33:15.155958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:33:15.155969 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:33:15.157717 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:33:15.158280 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:33:15.158282 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:33:15.158282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:33:15.158355 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:33:15.158639 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:33:15.158640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:33:15.158640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:33:15.158795 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:33:15.158876 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 23:33:15.159215 (+ 339us) server_negotiation.cc:299] Negotiation successful
0618 23:33:15.159320 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260618 23:33:15.160097 11829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:33:15.148436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40635 (local address 127.0.98.1:58531)
0618 23:33:15.148668 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0618 23:33:15.148737 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0618 23:33:15.149590 (+ 853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:33:15.149915 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:33:15.149920 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:33:15.150231 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:33:15.150600 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:33:15.150608 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.151727 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.151729 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:33:15.152871 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:33:15.152874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.153318 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.153320 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:33:15.153478 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:33:15.154160 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:33:15.154171 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:33:15.155828 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:33:15.157852 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:33:15.157863 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:33:15.157867 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:33:15.158176 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:33:15.158427 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:33:15.158428 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:33:15.158428 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:33:15.158568 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:33:15.158864 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:33:15.158871 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:33:15.159071 (+ 200us) client_negotiation.cc:769] Sending connection context
0618 23:33:15.159196 (+ 125us) client_negotiation.cc:240] Negotiation successful
0618 23:33:15.159890 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260618 23:33:15.160689 11826 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40635
I20260618 23:33:15.160813 11826 heartbeater.cc:464] Registering TS with master...
I20260618 23:33:15.161063 11826 heartbeater.cc:511] Master 127.0.98.62:40635 requested a full tablet report, sending...
I20260618 23:33:15.161791 11614 ts_manager.cc:194] Registered new tserver with Master: 934655bb12d64e46a999167c40d27b70 (127.0.98.1:43059)
I20260618 23:33:15.162575 11614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260618 23:33:15.183116 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46251/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40635
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 23:33:15 dist-test-slave-6224 krb5kdc[11138](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781825595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260618 23:33:15.312863 11832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:33:15.313131 11832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:33:15.313175 11832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:33:15.315116 11832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:33:15.315196 11832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:33:15.315244 11832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:33:15.317209 11832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:46251/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:33:15.317534 11832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:33:15.317817 11832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:33:15.324275 11842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:33:15.325167 11840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:33:15.325256 11839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:33:15.326423 11832 server_base.cc:1034] running on GCE node
I20260618 23:33:15.326731 11832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:33:15.327064 11832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:33:15.328248 11832 hybrid_clock.cc:648] HybridClock initialized: now 1781825595328221 us; error 41 us; skew 500 ppm
Jun 18 23:33:15 dist-test-slave-6224 krb5kdc[11138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:33:15.331760 11832 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:33:15.332324 11832 webserver.cc:460] Webserver started at http://127.0.98.2:42357/ using document root <none> and password file <none>
I20260618 23:33:15.332628 11832 fs_manager.cc:359] Metadata directory not provided
I20260618 23:33:15.332706 11832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:33:15.332857 11832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 23:33:15 dist-test-slave-6224 krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260618 23:33:15.349385 11832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "c1e4b9538582457daf485a2f92e189f3"
format_stamp: "Formatted at 2026-06-18 23:33:15 on dist-test-slave-6224"
server_key: "270eff6d00640239e60408180cbfa0cb"
server_key_iv: "1bb5e19ee5cdcc7de4857629cd50b101"
server_key_version: "kuduclusterkey@0"
I20260618 23:33:15.349881 11832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c1e4b9538582457daf485a2f92e189f3"
format_stamp: "Formatted at 2026-06-18 23:33:15 on dist-test-slave-6224"
server_key: "270eff6d00640239e60408180cbfa0cb"
server_key_iv: "1bb5e19ee5cdcc7de4857629cd50b101"
server_key_version: "kuduclusterkey@0"
I20260618 23:33:15.354837 11832 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260618 23:33:15.368201 11849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:33:15.368484 11832 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:33:15.368582 11832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "c1e4b9538582457daf485a2f92e189f3"
format_stamp: "Formatted at 2026-06-18 23:33:15 on dist-test-slave-6224"
server_key: "270eff6d00640239e60408180cbfa0cb"
server_key_iv: "1bb5e19ee5cdcc7de4857629cd50b101"
server_key_version: "kuduclusterkey@0"
I20260618 23:33:15.368685 11832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:33:15.392479 11832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:33:15.396258 11832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:33:15.396435 11832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:33:15.396785 11832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:33:15.397172 11832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:33:15.397225 11832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:33:15.397258 11832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:33:15.397276 11832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:33:15.404474 11832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45053
I20260618 23:33:15.405017 11832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:33:15.406555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 11832
I20260618 23:33:15.406664 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:33:15.406840 392 external_mini_cluster.cc:1428] Setting key 0d24d5472a4e2813cc2e223226958ae1
I20260618 23:33:15.416378 11963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45053 every 8 connection(s)
Jun 18 23:33:15 dist-test-slave-6224 krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260618 23:33:15.439486 11828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:33:15.420650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51425 (local address 127.0.98.62:40635)
0618 23:33:15.420718 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0618 23:33:15.420721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:33:15.423921 (+ 3200us) server_negotiation.cc:407] Connection header received
0618 23:33:15.424703 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:33:15.424705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:33:15.424727 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:33:15.424759 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:33:15.425668 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.426464 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.428271 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.428651 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.428706 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:33:15.431746 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:33:15.431748 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:33:15.431748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:33:15.431758 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:33:15.433718 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:33:15.438204 (+ 4486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:33:15.438206 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:33:15.438206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:33:15.438289 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:33:15.438640 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:33:15.438642 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:33:15.438642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:33:15.438801 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:33:15.438860 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0618 23:33:15.439272 (+ 412us) server_negotiation.cc:299] Negotiation successful
0618 23:33:15.439380 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260618 23:33:15.441442 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:33:15.420607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40635 (local address 127.0.98.2:51425)
0618 23:33:15.423720 (+ 3113us) negotiation.cc:107] Waiting for socket to connect
0618 23:33:15.423789 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0618 23:33:15.424565 (+ 776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:33:15.424959 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:33:15.424963 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:33:15.425240 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:33:15.425551 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:33:15.425558 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.426589 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.426590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:33:15.428143 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:33:15.428148 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.428821 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.428822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:33:15.428988 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:33:15.429710 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:33:15.429721 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:33:15.431598 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:33:15.433845 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:33:15.433857 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:33:15.433860 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:33:15.438045 (+ 4185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:33:15.438365 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:33:15.438366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:33:15.438366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:33:15.438550 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:33:15.438896 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:33:15.438903 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:33:15.439109 (+ 206us) client_negotiation.cc:769] Sending connection context
0618 23:33:15.440912 (+ 1803us) client_negotiation.cc:240] Negotiation successful
0618 23:33:15.441118 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3048,"thread_start_us":122,"threads_started":1}
I20260618 23:33:15.442219 11964 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40635
I20260618 23:33:15.442320 11964 heartbeater.cc:464] Registering TS with master...
I20260618 23:33:15.442550 11964 heartbeater.cc:511] Master 127.0.98.62:40635 requested a full tablet report, sending...
I20260618 23:33:15.443039 11614 ts_manager.cc:194] Registered new tserver with Master: c1e4b9538582457daf485a2f92e189f3 (127.0.98.2:45053)
I20260618 23:33:15.443620 11614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260618 23:33:15.476457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:46251/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40635
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:33:15.606868 11969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:33:15.607125 11969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:33:15.607254 11969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:33:15.608853 11969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:33:15.608939 11969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:33:15.608990 11969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:33:15.610893 11969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:46251/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:33:15.611212 11969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:33:15.611472 11969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:33:15.616901 11975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:33:15.617202 11974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:33:15.617349 11977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:33:15.617616 11969 server_base.cc:1034] running on GCE node
I20260618 23:33:15.617830 11969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:33:15.618141 11969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:33:15.619307 11969 hybrid_clock.cc:648] HybridClock initialized: now 1781825595619285 us; error 38 us; skew 500 ppm
Jun 18 23:33:15 dist-test-slave-6224 krb5kdc[11138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:33:15.622756 11969 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:33:15.623365 11969 webserver.cc:460] Webserver started at http://127.0.98.3:44543/ using document root <none> and password file <none>
I20260618 23:33:15.623677 11969 fs_manager.cc:359] Metadata directory not provided
I20260618 23:33:15.623736 11969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:33:15.623904 11969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 23:33:15 dist-test-slave-6224 krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260618 23:33:15.637398 11969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "60b0f769423443499befd9d4ac3da45b"
format_stamp: "Formatted at 2026-06-18 23:33:15 on dist-test-slave-6224"
server_key: "828cb75d694468ff60083439ff5d3812"
server_key_iv: "3baa8c3a13b33c2193f01d902ab18e18"
server_key_version: "kuduclusterkey@0"
I20260618 23:33:15.638113 11969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "60b0f769423443499befd9d4ac3da45b"
format_stamp: "Formatted at 2026-06-18 23:33:15 on dist-test-slave-6224"
server_key: "828cb75d694468ff60083439ff5d3812"
server_key_iv: "3baa8c3a13b33c2193f01d902ab18e18"
server_key_version: "kuduclusterkey@0"
I20260618 23:33:15.640249 11969 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260618 23:33:15.657280 11984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:33:15.657497 11969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:33:15.657603 11969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "60b0f769423443499befd9d4ac3da45b"
format_stamp: "Formatted at 2026-06-18 23:33:15 on dist-test-slave-6224"
server_key: "828cb75d694468ff60083439ff5d3812"
server_key_iv: "3baa8c3a13b33c2193f01d902ab18e18"
server_key_version: "kuduclusterkey@0"
I20260618 23:33:15.657722 11969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:33:15.681306 11969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:33:15.684528 11969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:33:15.685114 11969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:33:15.685446 11969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:33:15.685976 11969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:33:15.686044 11969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:33:15.686081 11969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:33:15.686103 11969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:33:15.696206 11969 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44847
I20260618 23:33:15.696733 11969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:33:15.701759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 11969
I20260618 23:33:15.701905 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:33:15.702122 392 external_mini_cluster.cc:1428] Setting key a8a69d77436e42d54a221e13d5771238
I20260618 23:33:15.705613 12097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44847 every 8 connection(s)
Jun 18 23:33:15 dist-test-slave-6224 krb5kdc[11138](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:33:15.714699 11828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:33:15.699130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60191 (local address 127.0.98.62:40635)
0618 23:33:15.699183 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0618 23:33:15.699185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:33:15.703994 (+ 4809us) server_negotiation.cc:407] Connection header received
0618 23:33:15.705285 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:33:15.705286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:33:15.705298 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:33:15.705352 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:33:15.706231 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.706789 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.708141 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.708388 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.708432 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:33:15.711208 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:33:15.711210 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:33:15.711210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:33:15.711218 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:33:15.712998 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:33:15.713550 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:33:15.713551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:33:15.713552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:33:15.713611 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:33:15.713904 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:33:15.713905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:33:15.713905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:33:15.714044 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:33:15.714090 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0618 23:33:15.714490 (+ 400us) server_negotiation.cc:299] Negotiation successful
0618 23:33:15.714595 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:33:15.715394 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:33:15.698344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40635 (local address 127.0.98.3:60191)
0618 23:33:15.703712 (+ 5368us) negotiation.cc:107] Waiting for socket to connect
0618 23:33:15.703790 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0618 23:33:15.705146 (+ 1356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:33:15.705503 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:33:15.705508 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:33:15.705810 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:33:15.706141 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:33:15.706150 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.706886 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.706888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:33:15.708034 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:33:15.708038 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.708544 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.708544 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:33:15.708698 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:33:15.709370 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:33:15.709381 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:33:15.711079 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:33:15.713111 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:33:15.713119 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:33:15.713122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:33:15.713438 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:33:15.713683 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:33:15.713684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:33:15.713684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:33:15.713823 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:33:15.714141 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:33:15.714147 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:33:15.714333 (+ 186us) client_negotiation.cc:769] Sending connection context
0618 23:33:15.714822 (+ 489us) client_negotiation.cc:240] Negotiation successful
0618 23:33:15.715117 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5253,"thread_start_us":581,"threads_started":1}
I20260618 23:33:15.716054 12098 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40635
I20260618 23:33:15.716157 12098 heartbeater.cc:464] Registering TS with master...
I20260618 23:33:15.716348 12098 heartbeater.cc:511] Master 127.0.98.62:40635 requested a full tablet report, sending...
I20260618 23:33:15.716737 11614 ts_manager.cc:194] Registered new tserver with Master: 60b0f769423443499befd9d4ac3da45b (127.0.98.3:44847)
I20260618 23:33:15.717126 11614 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60191
I20260618 23:33:15.726795 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:33:15.736569 11828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:33:15.728639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39876 (local address 127.0.98.62:40635)
0618 23:33:15.728690 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0618 23:33:15.728693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:33:15.728825 (+ 132us) server_negotiation.cc:407] Connection header received
0618 23:33:15.729303 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:33:15.729304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:33:15.729316 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:33:15.729384 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:33:15.729980 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.730483 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.731233 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.731478 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.731542 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:33:15.733471 (+ 1929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:33:15.733472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:33:15.733473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:33:15.733480 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:33:15.735186 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:33:15.735640 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:33:15.735641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:33:15.735641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:33:15.735709 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:33:15.735908 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:33:15.735909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:33:15.735909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:33:15.736050 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:33:15.736125 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 23:33:15.736388 (+ 263us) server_negotiation.cc:299] Negotiation successful
0618 23:33:15.736469 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260618 23:33:15.738938 11614 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:33:15.741086 11614 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:33:15.749372 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:33:15.744113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51288 (local address 127.0.98.1:43059)
0618 23:33:15.744495 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0618 23:33:15.744501 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:33:15.744714 (+ 213us) server_negotiation.cc:407] Connection header received
0618 23:33:15.744754 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:33:15.744756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:33:15.744774 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:33:15.744843 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:33:15.745094 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.746246 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.747489 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.748317 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.748391 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:33:15.748445 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0618 23:33:15.749248 (+ 803us) server_negotiation.cc:299] Negotiation successful
0618 23:33:15.749294 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":62,"threads_started":1}
I20260618 23:33:15.749784 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:33:15.743837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43059 (local address 127.0.0.1:51288)
0618 23:33:15.744281 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0618 23:33:15.744553 (+ 272us) client_negotiation.cc:174] Beginning negotiation
0618 23:33:15.744627 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:33:15.744922 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:33:15.744924 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:33:15.744926 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:33:15.745009 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:33:15.745012 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.746339 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.746340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:33:15.747395 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:33:15.747398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.748477 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.748477 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:33:15.748571 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:33:15.748584 (+ 13us) client_negotiation.cc:769] Sending connection context
0618 23:33:15.748647 (+ 63us) client_negotiation.cc:240] Negotiation successful
0618 23:33:15.748698 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260618 23:33:15.749951 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:33:15.744352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47534 (local address 127.0.98.2:45053)
0618 23:33:15.745031 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0618 23:33:15.745035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:33:15.745044 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:33:15.745081 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:33:15.745083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:33:15.745098 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:33:15.745166 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:33:15.745355 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.746379 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.748678 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.749410 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.749478 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:33:15.749548 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 23:33:15.749822 (+ 274us) server_negotiation.cc:299] Negotiation successful
0618 23:33:15.749871 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":68,"threads_started":1}
I20260618 23:33:15.750763 11760 tablet_service.cc:1467] Processing CreateTablet for tablet 88a450fc37d141db875de0273cda49dc (DEFAULT_TABLE table=test-table [id=b613dd7a50c04e0bbac88886268b7fd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:33:15.751142 11760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88a450fc37d141db875de0273cda49dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:33:15.753976 12111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:33:15.744056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44847 (local address 127.0.0.1:38478)
0618 23:33:15.746778 (+ 2722us) negotiation.cc:107] Waiting for socket to connect
0618 23:33:15.746788 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:33:15.746829 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:33:15.750496 (+ 3667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:33:15.750496 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:33:15.750499 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:33:15.750534 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:33:15.750536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.751709 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.751710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:33:15.752727 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:33:15.752730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.753749 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.753750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:33:15.753809 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:33:15.753813 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:33:15.753864 (+ 51us) client_negotiation.cc:240] Negotiation successful
0618 23:33:15.753902 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2704,"thread_start_us":57,"threads_started":1}
I20260618 23:33:15.754504 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:33:15.744328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38478 (local address 127.0.98.3:44847)
0618 23:33:15.750271 (+ 5943us) server_negotiation.cc:206] Beginning negotiation
0618 23:33:15.750276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:33:15.750286 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:33:15.750325 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:33:15.750327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:33:15.750342 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:33:15.750414 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:33:15.750608 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.751615 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.752845 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.753573 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.753646 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:33:15.753716 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 23:33:15.754385 (+ 669us) server_negotiation.cc:299] Negotiation successful
0618 23:33:15.754424 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5904,"thread_start_us":70,"threads_started":1}
I20260618 23:33:15.750000 12112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:33:15.743837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45053 (local address 127.0.0.1:47534)
0618 23:33:15.744514 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0618 23:33:15.744552 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0618 23:33:15.744912 (+ 360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:33:15.745230 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:33:15.745230 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:33:15.745232 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:33:15.745280 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:33:15.745282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.746457 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.746458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:33:15.748567 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:33:15.748568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.749579 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.749580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:33:15.749645 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:33:15.749648 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:33:15.749695 (+ 47us) client_negotiation.cc:240] Negotiation successful
0618 23:33:15.749738 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"mutex_wait_us":1035,"thread_start_us":85,"threads_started":1}
I20260618 23:33:15.755260 12032 tablet_service.cc:1467] Processing CreateTablet for tablet 88a450fc37d141db875de0273cda49dc (DEFAULT_TABLE table=test-table [id=b613dd7a50c04e0bbac88886268b7fd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:33:15.755631 12032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88a450fc37d141db875de0273cda49dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:33:15.755775 12116 tablet_bootstrap.cc:492] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70: Bootstrap starting.
I20260618 23:33:15.756904 12116 tablet_bootstrap.cc:654] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70: Neither blocks nor log segments found. Creating new log.
I20260618 23:33:15.757304 12116 log.cc:824] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70: Log is configured to *not* fsync() on all Append() calls
I20260618 23:33:15.759193 12118 tablet_bootstrap.cc:492] T 88a450fc37d141db875de0273cda49dc P 60b0f769423443499befd9d4ac3da45b: Bootstrap starting.
I20260618 23:33:15.759814 11898 tablet_service.cc:1467] Processing CreateTablet for tablet 88a450fc37d141db875de0273cda49dc (DEFAULT_TABLE table=test-table [id=b613dd7a50c04e0bbac88886268b7fd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:33:15.760155 11898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88a450fc37d141db875de0273cda49dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:33:15.760221 12118 tablet_bootstrap.cc:654] T 88a450fc37d141db875de0273cda49dc P 60b0f769423443499befd9d4ac3da45b: Neither blocks nor log segments found. Creating new log.
I20260618 23:33:15.760514 12118 log.cc:824] T 88a450fc37d141db875de0273cda49dc P 60b0f769423443499befd9d4ac3da45b: Log is configured to *not* fsync() on all Append() calls
I20260618 23:33:15.761257 12118 tablet_bootstrap.cc:492] T 88a450fc37d141db875de0273cda49dc P 60b0f769423443499befd9d4ac3da45b: No bootstrap required, opened a new log
I20260618 23:33:15.761318 12118 ts_tablet_manager.cc:1397] T 88a450fc37d141db875de0273cda49dc P 60b0f769423443499befd9d4ac3da45b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260618 23:33:15.763367 12118 raft_consensus.cc:348] T 88a450fc37d141db875de0273cda49dc P 60b0f769423443499befd9d4ac3da45b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934655bb12d64e46a999167c40d27b70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43059 } } peers { permanent_uuid: "c1e4b9538582457daf485a2f92e189f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45053 } } peers { permanent_uuid: "60b0f769423443499befd9d4ac3da45b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44847 } }
I20260618 23:33:15.764032 12118 raft_consensus.cc:374] T 88a450fc37d141db875de0273cda49dc P 60b0f769423443499befd9d4ac3da45b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:33:15.764087 12118 raft_consensus.cc:729] T 88a450fc37d141db875de0273cda49dc P 60b0f769423443499befd9d4ac3da45b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b0f769423443499befd9d4ac3da45b, State: Initialized, Role: FOLLOWER
I20260618 23:33:15.764204 12118 consensus_queue.cc:260] T 88a450fc37d141db875de0273cda49dc P 60b0f769423443499befd9d4ac3da45b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934655bb12d64e46a999167c40d27b70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43059 } } peers { permanent_uuid: "c1e4b9538582457daf485a2f92e189f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45053 } } peers { permanent_uuid: "60b0f769423443499befd9d4ac3da45b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44847 } }
I20260618 23:33:15.764484 12118 ts_tablet_manager.cc:1428] T 88a450fc37d141db875de0273cda49dc P 60b0f769423443499befd9d4ac3da45b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260618 23:33:15.764664 12098 heartbeater.cc:503] Master 127.0.98.62:40635 was elected leader, sending a full tablet report...
I20260618 23:33:15.765054 12121 tablet_bootstrap.cc:492] T 88a450fc37d141db875de0273cda49dc P c1e4b9538582457daf485a2f92e189f3: Bootstrap starting.
I20260618 23:33:15.766103 12121 tablet_bootstrap.cc:654] T 88a450fc37d141db875de0273cda49dc P c1e4b9538582457daf485a2f92e189f3: Neither blocks nor log segments found. Creating new log.
I20260618 23:33:15.766425 12121 log.cc:824] T 88a450fc37d141db875de0273cda49dc P c1e4b9538582457daf485a2f92e189f3: Log is configured to *not* fsync() on all Append() calls
I20260618 23:33:15.767251 12121 tablet_bootstrap.cc:492] T 88a450fc37d141db875de0273cda49dc P c1e4b9538582457daf485a2f92e189f3: No bootstrap required, opened a new log
I20260618 23:33:15.767252 12116 tablet_bootstrap.cc:492] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70: No bootstrap required, opened a new log
I20260618 23:33:15.767339 12116 ts_tablet_manager.cc:1397] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260618 23:33:15.767372 12121 ts_tablet_manager.cc:1397] T 88a450fc37d141db875de0273cda49dc P c1e4b9538582457daf485a2f92e189f3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:33:15.770164 12121 raft_consensus.cc:348] T 88a450fc37d141db875de0273cda49dc P c1e4b9538582457daf485a2f92e189f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934655bb12d64e46a999167c40d27b70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43059 } } peers { permanent_uuid: "c1e4b9538582457daf485a2f92e189f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45053 } } peers { permanent_uuid: "60b0f769423443499befd9d4ac3da45b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44847 } }
I20260618 23:33:15.770444 12121 raft_consensus.cc:374] T 88a450fc37d141db875de0273cda49dc P c1e4b9538582457daf485a2f92e189f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:33:15.770534 12121 raft_consensus.cc:729] T 88a450fc37d141db875de0273cda49dc P c1e4b9538582457daf485a2f92e189f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e4b9538582457daf485a2f92e189f3, State: Initialized, Role: FOLLOWER
I20260618 23:33:15.770530 12116 raft_consensus.cc:348] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934655bb12d64e46a999167c40d27b70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43059 } } peers { permanent_uuid: "c1e4b9538582457daf485a2f92e189f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45053 } } peers { permanent_uuid: "60b0f769423443499befd9d4ac3da45b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44847 } }
I20260618 23:33:15.770681 12116 raft_consensus.cc:374] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:33:15.770723 12116 raft_consensus.cc:729] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 934655bb12d64e46a999167c40d27b70, State: Initialized, Role: FOLLOWER
I20260618 23:33:15.770833 12116 consensus_queue.cc:260] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934655bb12d64e46a999167c40d27b70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43059 } } peers { permanent_uuid: "c1e4b9538582457daf485a2f92e189f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45053 } } peers { permanent_uuid: "60b0f769423443499befd9d4ac3da45b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44847 } }
I20260618 23:33:15.770833 12121 consensus_queue.cc:260] T 88a450fc37d141db875de0273cda49dc P c1e4b9538582457daf485a2f92e189f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934655bb12d64e46a999167c40d27b70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43059 } } peers { permanent_uuid: "c1e4b9538582457daf485a2f92e189f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45053 } } peers { permanent_uuid: "60b0f769423443499befd9d4ac3da45b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44847 } }
I20260618 23:33:15.771142 12121 ts_tablet_manager.cc:1428] T 88a450fc37d141db875de0273cda49dc P c1e4b9538582457daf485a2f92e189f3: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260618 23:33:15.771142 12116 ts_tablet_manager.cc:1428] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260618 23:33:15.771322 11964 heartbeater.cc:503] Master 127.0.98.62:40635 was elected leader, sending a full tablet report...
I20260618 23:33:15.771374 11826 heartbeater.cc:503] Master 127.0.98.62:40635 was elected leader, sending a full tablet report...
I20260618 23:33:15.799333 12123 raft_consensus.cc:482] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:33:15.799597 12123 raft_consensus.cc:504] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934655bb12d64e46a999167c40d27b70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43059 } } peers { permanent_uuid: "c1e4b9538582457daf485a2f92e189f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45053 } } peers { permanent_uuid: "60b0f769423443499befd9d4ac3da45b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44847 } }
I20260618 23:33:15.800122 12123 leader_election.cc:288] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c1e4b9538582457daf485a2f92e189f3 (127.0.98.2:45053), 60b0f769423443499befd9d4ac3da45b (127.0.98.3:44847)
I20260618 23:33:15.804802 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:33:15.800501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56627 (local address 127.0.98.2:45053)
0618 23:33:15.800562 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0618 23:33:15.800564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:33:15.801085 (+ 521us) server_negotiation.cc:407] Connection header received
0618 23:33:15.801205 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:33:15.801206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:33:15.801222 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:33:15.801256 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:33:15.801558 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.802270 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.803873 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.804381 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.804434 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:33:15.804447 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0618 23:33:15.804679 (+ 232us) server_negotiation.cc:299] Negotiation successful
0618 23:33:15.804722 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260618 23:33:15.805755 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:33:15.800366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45053 (local address 127.0.98.1:56627)
0618 23:33:15.801001 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0618 23:33:15.801014 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:33:15.801123 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:33:15.801346 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:33:15.801348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:33:15.801351 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:33:15.801444 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:33:15.801447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.802426 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.802427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:33:15.803765 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:33:15.803768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.804524 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.804525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:33:15.804595 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:33:15.804599 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:33:15.805166 (+ 567us) client_negotiation.cc:240] Negotiation successful
0618 23:33:15.805226 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":129,"threads_started":1}
I20260618 23:33:15.806280 11918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a450fc37d141db875de0273cda49dc" candidate_uuid: "934655bb12d64e46a999167c40d27b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e4b9538582457daf485a2f92e189f3" is_pre_election: true
I20260618 23:33:15.806463 11918 raft_consensus.cc:2454] T 88a450fc37d141db875de0273cda49dc P c1e4b9538582457daf485a2f92e189f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 934655bb12d64e46a999167c40d27b70 in term 0.
I20260618 23:33:15.806828 11716 leader_election.cc:302] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 934655bb12d64e46a999167c40d27b70, c1e4b9538582457daf485a2f92e189f3; no voters:
I20260618 23:33:15.806973 12123 raft_consensus.cc:2781] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:33:15.807019 12123 raft_consensus.cc:482] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:33:15.807049 12123 raft_consensus.cc:3037] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:33:15.807091 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:33:15.800810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41411 (local address 127.0.98.3:44847)
0618 23:33:15.800857 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0618 23:33:15.800859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:33:15.801318 (+ 459us) server_negotiation.cc:407] Connection header received
0618 23:33:15.801460 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:33:15.801462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:33:15.801479 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:33:15.801513 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:33:15.803363 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.804183 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.806071 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.806620 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.806674 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:33:15.806689 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0618 23:33:15.806945 (+ 256us) server_negotiation.cc:299] Negotiation successful
0618 23:33:15.806998 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:33:15.807438 12126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:33:15.800622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44847 (local address 127.0.98.1:41411)
0618 23:33:15.801185 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0618 23:33:15.801200 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 23:33:15.801360 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:33:15.801599 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:33:15.801600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:33:15.801602 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:33:15.803222 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:33:15.803227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.804799 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.804800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:33:15.805949 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:33:15.805952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.806769 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.806770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:33:15.806849 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:33:15.806853 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:33:15.807250 (+ 397us) client_negotiation.cc:240] Negotiation successful
0618 23:33:15.807305 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":85,"threads_started":1}
I20260618 23:33:15.807847 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a450fc37d141db875de0273cda49dc" candidate_uuid: "934655bb12d64e46a999167c40d27b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b0f769423443499befd9d4ac3da45b" is_pre_election: true
I20260618 23:33:15.808039 12043 raft_consensus.cc:2454] T 88a450fc37d141db875de0273cda49dc P 60b0f769423443499befd9d4ac3da45b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 934655bb12d64e46a999167c40d27b70 in term 0.
I20260618 23:33:15.808131 12123 raft_consensus.cc:504] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934655bb12d64e46a999167c40d27b70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43059 } } peers { permanent_uuid: "c1e4b9538582457daf485a2f92e189f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45053 } } peers { permanent_uuid: "60b0f769423443499befd9d4ac3da45b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44847 } }
I20260618 23:33:15.808279 12123 leader_election.cc:288] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70 [CANDIDATE]: Term 1 election: Requested vote from peers c1e4b9538582457daf485a2f92e189f3 (127.0.98.2:45053), 60b0f769423443499befd9d4ac3da45b (127.0.98.3:44847)
I20260618 23:33:15.808557 11918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a450fc37d141db875de0273cda49dc" candidate_uuid: "934655bb12d64e46a999167c40d27b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1e4b9538582457daf485a2f92e189f3"
I20260618 23:33:15.808672 11918 raft_consensus.cc:3037] T 88a450fc37d141db875de0273cda49dc P c1e4b9538582457daf485a2f92e189f3 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:33:15.809330 12043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a450fc37d141db875de0273cda49dc" candidate_uuid: "934655bb12d64e46a999167c40d27b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b0f769423443499befd9d4ac3da45b"
I20260618 23:33:15.809454 12043 raft_consensus.cc:3037] T 88a450fc37d141db875de0273cda49dc P 60b0f769423443499befd9d4ac3da45b [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:33:15.809896 11918 raft_consensus.cc:2454] T 88a450fc37d141db875de0273cda49dc P c1e4b9538582457daf485a2f92e189f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 934655bb12d64e46a999167c40d27b70 in term 1.
I20260618 23:33:15.810110 11716 leader_election.cc:302] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 934655bb12d64e46a999167c40d27b70, c1e4b9538582457daf485a2f92e189f3; no voters:
I20260618 23:33:15.810212 12123 raft_consensus.cc:2781] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:33:15.810407 12123 raft_consensus.cc:686] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70 [term 1 LEADER]: Becoming Leader. State: Replica: 934655bb12d64e46a999167c40d27b70, State: Running, Role: LEADER
I20260618 23:33:15.810424 12043 raft_consensus.cc:2454] T 88a450fc37d141db875de0273cda49dc P 60b0f769423443499befd9d4ac3da45b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 934655bb12d64e46a999167c40d27b70 in term 1.
I20260618 23:33:15.810532 12123 consensus_queue.cc:237] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934655bb12d64e46a999167c40d27b70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43059 } } peers { permanent_uuid: "c1e4b9538582457daf485a2f92e189f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45053 } } peers { permanent_uuid: "60b0f769423443499befd9d4ac3da45b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44847 } }
I20260618 23:33:15.811704 11614 catalog_manager.cc:5495] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 934655bb12d64e46a999167c40d27b70 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "934655bb12d64e46a999167c40d27b70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "934655bb12d64e46a999167c40d27b70" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1e4b9538582457daf485a2f92e189f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60b0f769423443499befd9d4ac3da45b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44847 } health_report { overall_health: UNKNOWN } } }
I20260618 23:33:15.830687 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:33:15.827297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51296 (local address 127.0.98.1:43059)
0618 23:33:15.827347 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0618 23:33:15.827349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:33:15.827356 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:33:15.827398 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:33:15.827399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:33:15.827426 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:33:15.827502 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:33:15.827723 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.828486 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.829602 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:33:15.829881 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:33:15.829945 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:33:15.830154 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:33:15.830296 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:33:15.830371 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 23:33:15.830557 (+ 186us) server_negotiation.cc:299] Negotiation successful
0618 23:33:15.830579 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:33:15.833714 12043 raft_consensus.cc:1261] T 88a450fc37d141db875de0273cda49dc P 60b0f769423443499befd9d4ac3da45b [term 1 FOLLOWER]: Refusing update from remote peer 934655bb12d64e46a999167c40d27b70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:33:15.834118 12123 consensus_queue.cc:1035] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b0f769423443499befd9d4ac3da45b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:33:15.834478 11918 raft_consensus.cc:1261] T 88a450fc37d141db875de0273cda49dc P c1e4b9538582457daf485a2f92e189f3 [term 1 FOLLOWER]: Refusing update from remote peer 934655bb12d64e46a999167c40d27b70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:33:15.834947 12123 consensus_queue.cc:1035] T 88a450fc37d141db875de0273cda49dc P 934655bb12d64e46a999167c40d27b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1e4b9538582457daf485a2f92e189f3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:33:15.838948 12130 mvcc.cc:205] Tried to move back new op lower bound from 7298357640514588672 to 7298357640424853504. Current Snapshot: MvccSnapshot[applied={T|T < 7298357640514588672}]
I20260618 23:33:15.839695 12131 mvcc.cc:205] Tried to move back new op lower bound from 7298357640514588672 to 7298357640424853504. Current Snapshot: MvccSnapshot[applied={T|T < 7298357640514588672}]
I20260618 23:33:15.840840 12132 mvcc.cc:205] Tried to move back new op lower bound from 7298357640514588672 to 7298357640424853504. Current Snapshot: MvccSnapshot[applied={T|T < 7298357640514588672}]
I20260618 23:33:15.845975 11614 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:33:15.846127 11614 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:33:15.847354 11614 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 034bbef8728f487aa169891f03e3c2fe: Sending DeleteTablet for 3 replicas of tablet 88a450fc37d141db875de0273cda49dc
I20260618 23:33:15.848076 11898 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a450fc37d141db875de0273cda49dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:33:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47534
I20260618 23:33:15.848413 12032 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a450fc37d141db875de0273cda49dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:33:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38478
I20260618 23:33:15.848999 11760 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a450fc37d141db875de0273cda49dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:33:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51288
I20260618 23:33:15.850674 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 11695
I20260618 23:33:15.856676 12140 tablet_replica.cc:330] T 88a450fc37d141db875de0273cda49dc P 60b0f769423443499befd9d4ac3da45b: stopping tablet replica
I20260618 23:33:15.856889 12140 raft_consensus.cc:2229] T 88a450fc37d141db875de0273cda49dc P 60b0f769423443499befd9d4ac3da45b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:33:15.857053 12140 raft_consensus.cc:2258] T 88a450fc37d141db875de0273cda49dc P 60b0f769423443499befd9d4ac3da45b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:33:15.857268 12139 tablet_replica.cc:330] T 88a450fc37d141db875de0273cda49dc P c1e4b9538582457daf485a2f92e189f3: stopping tablet replica
I20260618 23:33:15.857383 12139 raft_consensus.cc:2229] T 88a450fc37d141db875de0273cda49dc P c1e4b9538582457daf485a2f92e189f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:33:15.857512 12139 raft_consensus.cc:2258] T 88a450fc37d141db875de0273cda49dc P c1e4b9538582457daf485a2f92e189f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:33:15.857887 12140 ts_tablet_manager.cc:1905] T 88a450fc37d141db875de0273cda49dc P 60b0f769423443499befd9d4ac3da45b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:33:15.860443 12140 ts_tablet_manager.cc:1918] T 88a450fc37d141db875de0273cda49dc P 60b0f769423443499befd9d4ac3da45b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:33:15.860615 12140 log.cc:1196] T 88a450fc37d141db875de0273cda49dc P 60b0f769423443499befd9d4ac3da45b: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/88a450fc37d141db875de0273cda49dc
I20260618 23:33:15.861541 12140 ts_tablet_manager.cc:1939] T 88a450fc37d141db875de0273cda49dc P 60b0f769423443499befd9d4ac3da45b: Deleting consensus metadata
I20260618 23:33:15.862300 11601 catalog_manager.cc:4841] TS 60b0f769423443499befd9d4ac3da45b (127.0.98.3:44847): tablet 88a450fc37d141db875de0273cda49dc (table test-table [id=b613dd7a50c04e0bbac88886268b7fd3]) successfully deleted
I20260618 23:33:15.862897 12139 ts_tablet_manager.cc:1905] T 88a450fc37d141db875de0273cda49dc P c1e4b9538582457daf485a2f92e189f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:33:15.864595 12139 ts_tablet_manager.cc:1918] T 88a450fc37d141db875de0273cda49dc P c1e4b9538582457daf485a2f92e189f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:33:15.864790 12139 log.cc:1196] T 88a450fc37d141db875de0273cda49dc P c1e4b9538582457daf485a2f92e189f3: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/88a450fc37d141db875de0273cda49dc
I20260618 23:33:15.865185 12139 ts_tablet_manager.cc:1939] T 88a450fc37d141db875de0273cda49dc P c1e4b9538582457daf485a2f92e189f3: Deleting consensus metadata
W20260618 23:33:15.865239 11599 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43059 (error 108)
I20260618 23:33:15.865775 11601 catalog_manager.cc:4841] TS c1e4b9538582457daf485a2f92e189f3 (127.0.98.2:45053): tablet 88a450fc37d141db875de0273cda49dc (table test-table [id=b613dd7a50c04e0bbac88886268b7fd3]) successfully deleted
I20260618 23:33:15.866212 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 11832
I20260618 23:33:15.866345 12110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:33:15.866201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43059 (local address 127.0.0.1:51300)
0618 23:33:15.866259 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0618 23:33:15.866299 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260618 23:33:15.867240 11599 catalog_manager.cc:4584] TS 934655bb12d64e46a999167c40d27b70 (127.0.98.1:43059): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 88a450fc37d141db875de0273cda49dc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43059: connect: Connection refused (error 111)
I20260618 23:33:15.873488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 11969
I20260618 23:33:15.881693 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 11582
I20260618 23:33:15.889230 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260618 23:33:16.236055 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260618 23:33:16.236202 392 mini_ranger.cc:66] Stopping Ranger...
I20260618 23:33:16.582639 392 mini_ranger.cc:68] Stopped Ranger
2026-06-18 23:33:16.582 UTC [11168] LOG: received smart shutdown request
2026-06-18 23:33:16.586 UTC [11168] LOG: background worker "logical replication launcher" (PID 11177) exited with exit code 1
2026-06-18 23:33:16.587 UTC [11172] LOG: shutting down
2026-06-18 23:33:16.804 UTC [11168] LOG: database system is shut down
2026-06-18T23:33:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48924 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:33:16 dist-test-slave-6224 krb5kdc[12151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:33:16 dist-test-slave-6224 krb5kdc[12151](info): set up 2 sockets
Jun 18 23:33:16 dist-test-slave-6224 krb5kdc[12151](info): commencing operation
krb5kdc: starting...
W20260618 23:33:17.992177 392 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:33:18 dist-test-slave-6224 krb5kdc[12151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:33:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:33:18Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260618 23:33:18.180712 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskcnfpmz/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/postgres -l logfile start
2026-06-18 23:33:21.014 UTC [12181] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-18 23:33:21.014 UTC [12181] LOG: listening on IPv4 address "127.0.98.20", port 35133
2026-06-18 23:33:21.019 UTC [12181] LOG: listening on Unix socket "/tmp/.s.PGSQL.35133"
2026-06-18 23:33:21.044 UTC [12184] LOG: database system was shut down at 2026-06-18 23:33:19 UTC
2026-06-18 23:33:21.048 UTC [12181] LOG: database system is ready to accept connections
I20260618 23:33:22.044332 392 mini_postgres.cc:96] Postgres bound to 35133
2026-06-18 23:33:22.051 UTC [12192] FATAL: database "slave" does not exist
127.0.98.20:35133 - accepting connections
I20260618 23:33:22.052389 392 mini_ranger.cc:161] Starting Ranger...
I20260618 23:33:22.072270 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260618 23:33:22.693012 392 mini_ranger.cc:87] Created ranger Postgres database
I20260618 23:33:22.693118 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ranger-admin
2026-06-18 23:33:22,995 [I] DB FLAVOR :POSTGRES
2026-06-18 23:33:22,996 [I] --------- Verifying Ranger DB connection ---------
2026-06-18 23:33:22,996 [I] Checking connection..
2026-06-18 23:33:22,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-18 23:33:23,307 [I] Checking connection passed.
2026-06-18 23:33:23,307 [I] --------- Verifying version history table ---------
2026-06-18 23:33:23,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 23:33:23,608 [I] Table x_db_version_h does not exist in database ranger
2026-06-18 23:33:23,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 23:33:23,895 [I] Table x_db_version_h does not exist in database ranger
2026-06-18 23:33:23,895 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-18 23:33:23,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-18 23:33:24.148 UTC [12256] WARNING: there is no transaction in progress
2026-06-18 23:33:24,156 [I] create_dbversion_catalog.sql file imported successfully
2026-06-18 23:33:24,156 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 23:33:24,470 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-18 23:33:24,471 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-18 23:33:24,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-18 23:33:24,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-18 23:33:25,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-6224.c.gcp-upstream.internal','N') ;"
2026-06-18 23:33:25,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-18 23:33:25,614 [I] Table x_portal_user does not exist in database ranger
2026-06-18 23:33:25,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-18 23:33:25,908 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-18 23:33:25,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-18 23:33:26,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-18 23:33:26,456 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-18 23:33:26,457 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-18 23:33:27.570 UTC [12364] WARNING: there is no transaction in progress
2026-06-18 23:33:27.591 UTC [12364] WARNING: there is no transaction in progress
2026-06-18 23:33:27.611 UTC [12364] WARNING: there is no transaction in progress
2026-06-18 23:33:27.631 UTC [12364] WARNING: there is no transaction in progress
2026-06-18 23:33:27.649 UTC [12364] WARNING: there is no transaction in progress
2026-06-18 23:33:27.672 UTC [12364] WARNING: there is no transaction in progress
2026-06-18 23:33:27.742 UTC [12364] WARNING: there is no transaction in progress
2026-06-18 23:33:27.750 UTC [12364] WARNING: there is no transaction in progress
2026-06-18 23:33:27.767 UTC [12364] WARNING: there is no transaction in progress
2026-06-18 23:33:27.779 UTC [12364] WARNING: there is no transaction in progress
2026-06-18 23:33:27.791 UTC [12364] WARNING: there is no transaction in progress
2026-06-18 23:33:27.807 UTC [12364] WARNING: there is no transaction in progress
2026-06-18 23:33:27.818 UTC [12364] WARNING: there is no transaction in progress
2026-06-18 23:33:27.825 UTC [12364] WARNING: there is no transaction in progress
2026-06-18 23:33:27.832 UTC [12364] WARNING: there is no transaction in progress
2026-06-18 23:33:27.839 UTC [12364] WARNING: there is no transaction in progress
2026-06-18 23:33:28,596 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-18 23:33:28,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-18 23:33:28,906 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-18 23:33:28,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-18 23:33:29,173 [I] Table x_portal_user already exists in database 'ranger'
2026-06-18 23:33:29,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-18 23:33:29,473 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-18 23:33:29,473 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-18 23:33:29,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-18 23:33:30,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-6224.c.gcp-upstream.internal';"
2026-06-18 23:33:30,301 [I] CORE_DB_SCHEMA import status has been updated
2026-06-18 23:33:30,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-18 23:33:30,550 [I] DB_PATCHES have already been applied
I20260618 23:33:30.557068 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskcnfpmz/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260618 23:33:30.557169 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260618 23:33:30.561035 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:38051
Jun 18, 2026 11:33:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 18, 2026 11:33:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 18, 2026 11:33:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 18, 2026 11:33:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 18, 2026 11:33:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 18, 2026 11:33:31 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 18, 2026 11:33:31 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260618 23:33:31.709003 392 mini_ranger.cc:160] Time spent starting Ranger: real 9.657s user 0.003s sys 0.007s
Jun 18, 2026 11:33:54 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 18, 2026 11:33:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 18, 2026 11:33:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 18, 2026 11:33:54 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 18, 2026 11:33:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 18, 2026 11:33:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 18, 2026 11:33:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 18, 2026 11:33:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 18, 2026 11:33:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 18, 2026 11:33:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 18, 2026 11:33:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 18, 2026 11:33:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 18, 2026 11:33:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 18, 2026 11:33:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 18, 2026 11:33:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 18, 2026 11:33:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 18, 2026 11:33:54 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 18, 2026 11:33:55 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260618 23:33:56.129346 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 18 23:33:56 dist-test-slave-6224 krb5kdc[12151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825636, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260618 23:33:56.215333 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260618 23:33:56.225804 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260618 23:33:57.096038 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260618 23:33:57.096140 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ranger-kms
I20260618 23:33:57.096158 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:35133
2026-06-18 23:33:57,317 [I] DB FLAVOR :POSTGRES
2026-06-18 23:33:57,318 [I] --------- Verifying Ranger DB connection ---------
2026-06-18 23:33:57,318 [I] Checking connection
2026-06-18 23:33:57,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-18 23:33:57,615 [I] connection success
2026-06-18 23:33:57,615 [I] --------- Verifying Ranger DB tables ---------
2026-06-18 23:33:57,615 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-18 23:33:57,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-18 23:33:57,895 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-18 23:33:57,895 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-18 23:33:57,895 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-18 23:33:57,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:35133/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-18 23:33:58,166 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260618 23:33:58.692307 392 mini_ranger_kms.cc:310] Created kms service
I20260618 23:33:58.892345 392 mini_ranger_kms.cc:326] Created kudu user
I20260618 23:33:58.950733 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 18 23:33:58 dist-test-slave-6224 krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825636, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260618 23:33:59.404999 392 mini_ranger_kms.cc:384] Added ranger policy
I20260618 23:33:59.405243 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskcnfpmz/build/release/bin/postgresql.jar:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskcnfpmz/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260618 23:33:59.405267 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260618 23:33:59.408578 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12545
I20260618 23:33:59.408668 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:42879
Jun 18, 2026 11:34:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 18, 2026 11:34:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 18, 2026 11:34:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 18, 2026 11:34:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 18, 2026 11:34:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 18, 2026 11:34:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260618 23:34:01.663657 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.448s user 0.000s sys 0.195s
I20260618 23:34:01.663754 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260618 23:34:01.663800 392 mini_ranger_kms.cc:401] 127.0.98.20:42879/kms/v1/keys
Jun 18 23:34:01 dist-test-slave-6224 krb5kdc[12151](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781825641, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 18 23:34:02 dist-test-slave-6224 krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825641, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 18 23:34:02 dist-test-slave-6224 krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825641, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 18 23:34:05 dist-test-slave-6224 krb5kdc[12151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:34:05.290673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35197
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35197
--ranger_config_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:42879/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:34:05.387902 12595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:05.388234 12595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:05.388262 12595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:05.389652 12595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:34:05.389706 12595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:05.389721 12595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:05.389735 12595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:34:05.389748 12595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:34:05.391687 12595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:42879/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35197
--ranger_config_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:05.391996 12595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:05.392290 12595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:05.398478 12600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:05.398463 12601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:05.398742 12595 server_base.cc:1034] running on GCE node
W20260618 23:34:05.398463 12603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:05.399166 12595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:05.399593 12595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:05.400787 12595 hybrid_clock.cc:648] HybridClock initialized: now 1781825645400759 us; error 54 us; skew 500 ppm
Jun 18 23:34:05 dist-test-slave-6224 krb5kdc[12151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:05.406978 12595 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:34:05.407759 12595 webserver.cc:460] Webserver started at http://127.0.98.62:46789/ using document root <none> and password file <none>
I20260618 23:34:05.408046 12595 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:05.408114 12595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:05.408300 12595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 23:34:05 dist-test-slave-6224 krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260618 23:34:05.479946 12595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "05ba038aa32d47a7a34d7a88eaae03c1"
format_stamp: "Formatted at 2026-06-18 23:34:05 on dist-test-slave-6224"
server_key: "98643215cfc2e571ca588c6c5671af07"
server_key_iv: "f455ce5dd1abaf48e70e970fdf094717"
server_key_version: "kuduclusterkey@0"
I20260618 23:34:05.480446 12595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "05ba038aa32d47a7a34d7a88eaae03c1"
format_stamp: "Formatted at 2026-06-18 23:34:05 on dist-test-slave-6224"
server_key: "98643215cfc2e571ca588c6c5671af07"
server_key_iv: "f455ce5dd1abaf48e70e970fdf094717"
server_key_version: "kuduclusterkey@0"
I20260618 23:34:05.482053 12595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:34:05.499982 12610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:05.500325 12595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:05.500506 12595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "05ba038aa32d47a7a34d7a88eaae03c1"
format_stamp: "Formatted at 2026-06-18 23:34:05 on dist-test-slave-6224"
server_key: "98643215cfc2e571ca588c6c5671af07"
server_key_iv: "f455ce5dd1abaf48e70e970fdf094717"
server_key_version: "kuduclusterkey@0"
I20260618 23:34:05.500635 12595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:05.524736 12595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:05.527259 12595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:05.527485 12595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:05.534049 12595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35197
I20260618 23:34:05.534183 12662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35197 every 8 connection(s)
I20260618 23:34:05.534719 12595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:34:05.535748 12663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:34:05.538806 12663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1: Bootstrap starting.
I20260618 23:34:05.539284 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 12595
I20260618 23:34:05.539431 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:34:05.539693 392 external_mini_cluster.cc:1428] Setting key b24e183fe5e8cf5be072a6467c5b852d
I20260618 23:34:05.539705 12663 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1: Neither blocks nor log segments found. Creating new log.
I20260618 23:34:05.540129 12663 log.cc:824] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:05.541363 12663 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1: No bootstrap required, opened a new log
I20260618 23:34:05.544683 12663 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ba038aa32d47a7a34d7a88eaae03c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35197 } }
I20260618 23:34:05.545002 12663 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:34:05.545073 12663 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ba038aa32d47a7a34d7a88eaae03c1, State: Initialized, Role: FOLLOWER
I20260618 23:34:05.545269 12663 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ba038aa32d47a7a34d7a88eaae03c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35197 } }
I20260618 23:34:05.545439 12663 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:34:05.545506 12663 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:34:05.545588 12663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:34:05.546578 12663 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ba038aa32d47a7a34d7a88eaae03c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35197 } }
I20260618 23:34:05.546756 12663 leader_election.cc:302] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05ba038aa32d47a7a34d7a88eaae03c1; no voters:
I20260618 23:34:05.547005 12663 leader_election.cc:288] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:34:05.547129 12668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:34:05.547364 12663 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:34:05.547405 12668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [term 1 LEADER]: Becoming Leader. State: Replica: 05ba038aa32d47a7a34d7a88eaae03c1, State: Running, Role: LEADER
I20260618 23:34:05.547613 12668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ba038aa32d47a7a34d7a88eaae03c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35197 } }
Jun 18 23:34:05 dist-test-slave-6224 krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:34:05.548063 12668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05ba038aa32d47a7a34d7a88eaae03c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ba038aa32d47a7a34d7a88eaae03c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35197 } } }
I20260618 23:34:05.548152 12668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [sys.catalog]: This master's current role is: LEADER
I20260618 23:34:05.548297 12668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05ba038aa32d47a7a34d7a88eaae03c1. Latest consensus state: current_term: 1 leader_uuid: "05ba038aa32d47a7a34d7a88eaae03c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ba038aa32d47a7a34d7a88eaae03c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35197 } } }
I20260618 23:34:05.548341 12668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [sys.catalog]: This master's current role is: LEADER
I20260618 23:34:05.548527 12677 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:34:05.549201 12677 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:34:05.549647 12663 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260618 23:34:05.554106 12677 catalog_manager.cc:1347] Generated new cluster ID: 601fda8234bc4d4586ff8343863289b6
I20260618 23:34:05.554294 12677 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:34:05.555827 12667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:05.540577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52242 (local address 127.0.98.62:35197)
0618 23:34:05.540812 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:05.540823 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:05.540863 (+ 40us) server_negotiation.cc:407] Connection header received
0618 23:34:05.541413 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:05.541422 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:05.541530 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:05.541747 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:05.542503 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:05.543491 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:05.544393 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:05.544875 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:05.545009 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:05.549481 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:05.549487 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:05.549487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:05.549519 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:05.553126 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:05.553771 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:05.553776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:05.553777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:05.553926 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:05.554164 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:05.554165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:05.554165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:05.554560 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:05.554750 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:05.555354 (+ 604us) server_negotiation.cc:299] Negotiation successful
0618 23:34:05.555515 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":82,"threads_started":1}
I20260618 23:34:05.585816 12677 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:34:05.586772 12677 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:34:05.606379 12677 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1: Generated new TSK 0
I20260618 23:34:05.606786 12677 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260618 23:34:06.411183 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:42879/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35197
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:34:06.531858 12708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:06.532264 12708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:06.532352 12708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:06.534482 12708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:06.534633 12708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:06.534734 12708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:34:06.537158 12708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:42879/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:06.537459 12708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:06.537832 12708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:06.545090 12718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:06.545101 12715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:06.545228 12716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:06.545550 12708 server_base.cc:1034] running on GCE node
I20260618 23:34:06.545820 12708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:06.546167 12708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:06.547379 12708 hybrid_clock.cc:648] HybridClock initialized: now 1781825646547353 us; error 49 us; skew 500 ppm
Jun 18 23:34:06 dist-test-slave-6224 krb5kdc[12151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:06.551384 12708 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:34:06.552186 12708 webserver.cc:460] Webserver started at http://127.0.98.1:37241/ using document root <none> and password file <none>
I20260618 23:34:06.552484 12708 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:06.552577 12708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:06.552827 12708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 23:34:06 dist-test-slave-6224 krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260618 23:34:06.571774 12708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "0882c9a09d3a408792782116d8943dcc"
format_stamp: "Formatted at 2026-06-18 23:34:06 on dist-test-slave-6224"
server_key: "6d8420e405c380edc35fcf4f27213342"
server_key_iv: "dcf90095eafc090157be67943e633e36"
server_key_version: "kuduclusterkey@0"
I20260618 23:34:06.573099 12708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0882c9a09d3a408792782116d8943dcc"
format_stamp: "Formatted at 2026-06-18 23:34:06 on dist-test-slave-6224"
server_key: "6d8420e405c380edc35fcf4f27213342"
server_key_iv: "dcf90095eafc090157be67943e633e36"
server_key_version: "kuduclusterkey@0"
I20260618 23:34:06.580003 12708 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.000s
I20260618 23:34:06.595829 12725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:06.596082 12708 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260618 23:34:06.596197 12708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "0882c9a09d3a408792782116d8943dcc"
format_stamp: "Formatted at 2026-06-18 23:34:06 on dist-test-slave-6224"
server_key: "6d8420e405c380edc35fcf4f27213342"
server_key_iv: "dcf90095eafc090157be67943e633e36"
server_key_version: "kuduclusterkey@0"
I20260618 23:34:06.596340 12708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:06.608191 12708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:06.611182 12708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:06.611479 12708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:06.611931 12708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:34:06.612432 12708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:34:06.612542 12708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:06.612696 12708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:34:06.612784 12708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:06.624239 12708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37345
I20260618 23:34:06.624883 12708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:34:06.633422 12838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37345 every 8 connection(s)
I20260618 23:34:06.634812 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 12708
I20260618 23:34:06.634943 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:34:06.635165 392 external_mini_cluster.cc:1428] Setting key 47ae0ace2fe9aac7e975e5650d0b1968
Jun 18 23:34:06 dist-test-slave-6224 krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260618 23:34:06.662411 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:06.638511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41717 (local address 127.0.98.62:35197)
0618 23:34:06.650013 (+ 11502us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:06.650016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:06.650028 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:34:06.650058 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:06.650059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:06.650075 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:06.650160 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:06.651299 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:06.652278 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:06.653994 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:06.654354 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:06.654408 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:06.657843 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:06.657845 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:06.657846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:06.657856 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:06.660258 (+ 2402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:06.660908 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:06.660910 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:06.660910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:06.660980 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:06.661277 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:06.661278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:06.661278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:06.661441 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:06.661489 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:06.662157 (+ 668us) server_negotiation.cc:299] Negotiation successful
0618 23:34:06.662283 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11461,"thread_start_us":69,"threads_started":1}
I20260618 23:34:06.663442 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:06.638056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35197 (local address 127.0.98.1:41717)
0618 23:34:06.647738 (+ 9682us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:06.647824 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:06.648582 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:06.650386 (+ 1804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:06.650394 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:06.650751 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:06.651118 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:06.651126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:06.652438 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:06.652439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:06.653868 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:06.653873 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:06.654551 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:06.654552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:06.654728 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:06.655536 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:06.655598 (+ 62us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:06.657664 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:06.660389 (+ 2725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:06.660405 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:06.660410 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:06.660813 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:06.661050 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:06.661051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:06.661051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:06.661201 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:06.661537 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:06.661544 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:06.661956 (+ 412us) client_negotiation.cc:769] Sending connection context
0618 23:34:06.662899 (+ 943us) client_negotiation.cc:240] Negotiation successful
0618 23:34:06.663135 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9598,"thread_start_us":173,"threads_started":1}
I20260618 23:34:06.664836 12839 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35197
I20260618 23:34:06.664978 12839 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:06.665225 12839 heartbeater.cc:511] Master 127.0.98.62:35197 requested a full tablet report, sending...
I20260618 23:34:06.666028 12627 ts_manager.cc:194] Registered new tserver with Master: 0882c9a09d3a408792782116d8943dcc (127.0.98.1:37345)
I20260618 23:34:06.667011 12627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260618 23:34:06.682808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:42879/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35197
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 23:34:06 dist-test-slave-6224 krb5kdc[12151](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781825646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260618 23:34:06.808688 12847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:06.809098 12847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:06.809182 12847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:06.811023 12847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:06.811100 12847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:06.811143 12847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:34:06.813436 12847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:42879/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:06.813887 12847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:06.814215 12847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:06.821986 12852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:06.822377 12853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:06.822717 12856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:06.823954 12847 server_base.cc:1034] running on GCE node
I20260618 23:34:06.824378 12847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:06.824760 12847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:06.825963 12847 hybrid_clock.cc:648] HybridClock initialized: now 1781825646825932 us; error 43 us; skew 500 ppm
Jun 18 23:34:06 dist-test-slave-6224 krb5kdc[12151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:06.829721 12847 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:34:06.830438 12847 webserver.cc:460] Webserver started at http://127.0.98.2:35637/ using document root <none> and password file <none>
I20260618 23:34:06.830799 12847 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:06.830873 12847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:06.831025 12847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 23:34:06 dist-test-slave-6224 krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260618 23:34:06.847509 12847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "1fa78cab745a413c9848b35668d9a903"
format_stamp: "Formatted at 2026-06-18 23:34:06 on dist-test-slave-6224"
server_key: "c4c5bc683059a2048779b94169a143d6"
server_key_iv: "770a689008fa7b16e44a87f1e7e741bf"
server_key_version: "kuduclusterkey@0"
I20260618 23:34:06.848410 12847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1fa78cab745a413c9848b35668d9a903"
format_stamp: "Formatted at 2026-06-18 23:34:06 on dist-test-slave-6224"
server_key: "c4c5bc683059a2048779b94169a143d6"
server_key_iv: "770a689008fa7b16e44a87f1e7e741bf"
server_key_version: "kuduclusterkey@0"
I20260618 23:34:06.854898 12847 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.002s
I20260618 23:34:06.882707 12863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:06.882957 12847 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260618 23:34:06.883069 12847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "1fa78cab745a413c9848b35668d9a903"
format_stamp: "Formatted at 2026-06-18 23:34:06 on dist-test-slave-6224"
server_key: "c4c5bc683059a2048779b94169a143d6"
server_key_iv: "770a689008fa7b16e44a87f1e7e741bf"
server_key_version: "kuduclusterkey@0"
I20260618 23:34:06.883215 12847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:06.899628 12847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:06.902348 12847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:06.902536 12847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:06.902875 12847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:34:06.903324 12847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:34:06.903378 12847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:06.903422 12847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:34:06.903446 12847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:06.914086 12847 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32849
I20260618 23:34:06.914645 12847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:34:06.917243 12976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32849 every 8 connection(s)
I20260618 23:34:06.921617 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 12847
I20260618 23:34:06.921751 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:34:06.922000 392 external_mini_cluster.cc:1428] Setting key eeef96421a73882ead53936b438b69fc
Jun 18 23:34:06 dist-test-slave-6224 krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:34:06.932230 12979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:06.916722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35197 (local address 127.0.98.2:38231)
0618 23:34:06.916936 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:06.916988 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:06.917787 (+ 799us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:06.918082 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:06.918086 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:06.918311 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:06.918537 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:06.918541 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:06.919302 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:06.919303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:06.920223 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:06.920225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:06.920668 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:06.920669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:06.920803 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:06.921423 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:06.921433 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:06.923221 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:06.925310 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:06.925318 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:06.925321 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:06.925638 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:06.925907 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:06.925910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:06.925911 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:06.926185 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:06.930342 (+ 4157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:06.930353 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:06.931409 (+ 1056us) client_negotiation.cc:769] Sending connection context
0618 23:34:06.931604 (+ 195us) client_negotiation.cc:240] Negotiation successful
0618 23:34:06.931865 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":114,"threads_started":1}
I20260618 23:34:06.932691 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:06.917087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38231 (local address 127.0.98.62:35197)
0618 23:34:06.917137 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:06.917140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:06.917153 (+ 13us) server_negotiation.cc:407] Connection header received
0618 23:34:06.917919 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:06.917920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:06.917934 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:06.917989 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:06.918645 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:06.919207 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:06.920325 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:06.920562 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:06.920644 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:06.923340 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:06.923342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:06.923343 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:06.923352 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:06.925203 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:06.925745 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:06.925747 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:06.925747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:06.925810 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:06.926308 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:06.926309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:06.926309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:06.926488 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:06.926557 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:06.932451 (+ 5894us) server_negotiation.cc:299] Negotiation successful
0618 23:34:06.932573 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260618 23:34:06.933179 12977 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35197
I20260618 23:34:06.933296 12977 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:06.933532 12977 heartbeater.cc:511] Master 127.0.98.62:35197 requested a full tablet report, sending...
I20260618 23:34:06.934015 12627 ts_manager.cc:194] Registered new tserver with Master: 1fa78cab745a413c9848b35668d9a903 (127.0.98.2:32849)
I20260618 23:34:06.934598 12627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38231
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260618 23:34:06.958673 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:42879/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35197
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:34:07.093034 12982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:07.093349 12982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:07.093396 12982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:07.095530 12982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:07.095824 12982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:07.095907 12982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:34:07.098654 12982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:42879/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:07.099153 12982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:07.099609 12982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:07.105806 12990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:07.105878 12987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:07.106370 12982 server_base.cc:1034] running on GCE node
W20260618 23:34:07.105988 12988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:07.106695 12982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:07.107019 12982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:07.108186 12982 hybrid_clock.cc:648] HybridClock initialized: now 1781825647108166 us; error 40 us; skew 500 ppm
Jun 18 23:34:07 dist-test-slave-6224 krb5kdc[12151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:07.111196 12982 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:34:07.111847 12982 webserver.cc:460] Webserver started at http://127.0.98.3:36393/ using document root <none> and password file <none>
I20260618 23:34:07.112150 12982 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:07.112205 12982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:07.112340 12982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 23:34:07 dist-test-slave-6224 krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260618 23:34:07.128749 12982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "33b0dfcf52554b55ba9876bf8a5f9403"
format_stamp: "Formatted at 2026-06-18 23:34:07 on dist-test-slave-6224"
server_key: "604148be007d4a1e40f531cc9e1dd5e2"
server_key_iv: "ea442b5954da657e301592cedef39480"
server_key_version: "kuduclusterkey@0"
I20260618 23:34:07.129305 12982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "33b0dfcf52554b55ba9876bf8a5f9403"
format_stamp: "Formatted at 2026-06-18 23:34:07 on dist-test-slave-6224"
server_key: "604148be007d4a1e40f531cc9e1dd5e2"
server_key_iv: "ea442b5954da657e301592cedef39480"
server_key_version: "kuduclusterkey@0"
I20260618 23:34:07.136511 12982 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260618 23:34:07.157246 12997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:07.157475 12982 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260618 23:34:07.157580 12982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "33b0dfcf52554b55ba9876bf8a5f9403"
format_stamp: "Formatted at 2026-06-18 23:34:07 on dist-test-slave-6224"
server_key: "604148be007d4a1e40f531cc9e1dd5e2"
server_key_iv: "ea442b5954da657e301592cedef39480"
server_key_version: "kuduclusterkey@0"
I20260618 23:34:07.157708 12982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:07.180400 12982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:07.183379 12982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:07.183610 12982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:07.183897 12982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:34:07.184391 12982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:34:07.184449 12982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:07.184494 12982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:34:07.184523 12982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:07.194855 12982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42513
I20260618 23:34:07.195386 12982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:34:07.197077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 12982
I20260618 23:34:07.197201 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:34:07.197285 13110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42513 every 8 connection(s)
I20260618 23:34:07.197422 392 external_mini_cluster.cc:1428] Setting key 4a6b62942a5760346adf1be6b437ffc8
Jun 18 23:34:07 dist-test-slave-6224 krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:34:07.211321 12843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:07.198886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57655 (local address 127.0.98.62:35197)
0618 23:34:07.198936 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:07.198938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:07.198953 (+ 15us) server_negotiation.cc:407] Connection header received
0618 23:34:07.199736 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:07.199738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:07.199751 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:07.199849 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:07.200759 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:07.201632 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:07.203447 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:07.203838 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:07.203885 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:07.207104 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:07.207106 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:07.207106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:07.207116 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:07.208978 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:07.210139 (+ 1161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:07.210140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:07.210141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:07.210203 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:07.210486 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:07.210487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:07.210487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:07.210650 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:07.210691 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:07.211106 (+ 415us) server_negotiation.cc:299] Negotiation successful
0618 23:34:07.211208 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260618 23:34:07.212188 13113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:07.198519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35197 (local address 127.0.98.3:57655)
0618 23:34:07.198782 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:07.198843 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:07.199605 (+ 762us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:07.199929 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:07.199935 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:07.200310 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:07.200638 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:07.200644 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:07.201777 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:07.201778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:07.203311 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:07.203315 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:07.204034 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:07.204035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:07.204207 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:07.204946 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:07.204958 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:07.206969 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:07.209702 (+ 2733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:07.209710 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:07.209714 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:07.210045 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:07.210276 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:07.210276 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:07.210277 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:07.210417 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:07.210737 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:07.210743 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:07.210947 (+ 204us) client_negotiation.cc:769] Sending connection context
0618 23:34:07.211724 (+ 777us) client_negotiation.cc:240] Negotiation successful
0618 23:34:07.211919 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":131,"threads_started":1}
I20260618 23:34:07.213086 13111 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35197
I20260618 23:34:07.213258 13111 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:07.213534 13111 heartbeater.cc:511] Master 127.0.98.62:35197 requested a full tablet report, sending...
I20260618 23:34:07.214133 12627 ts_manager.cc:194] Registered new tserver with Master: 33b0dfcf52554b55ba9876bf8a5f9403 (127.0.98.3:42513)
I20260618 23:34:07.214684 12627 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57655
I20260618 23:34:07.220448 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:34:07.221263 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 12708
I20260618 23:34:07.228437 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 12847
I20260618 23:34:07.234668 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 12982
I20260618 23:34:07.241423 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 12595
I20260618 23:34:07.248317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35197
--webserver_interface=127.0.98.62
--webserver_port=46789
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35197
--ranger_config_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:42879/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:42879/kms with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:34:07.334489 13114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:07.334723 13114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:07.334745 13114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:07.335850 13114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:34:07.335892 13114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:07.335906 13114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:07.335920 13114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:34:07.335933 13114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:34:07.337466 13114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:42879/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35197
--ranger_config_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=46789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:07.337658 13114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:07.337841 13114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:07.343001 13119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:07.343087 13114 server_base.cc:1034] running on GCE node
W20260618 23:34:07.343091 13120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:07.343187 13122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:07.343729 13114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:07.344080 13114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:07.345288 13114 hybrid_clock.cc:648] HybridClock initialized: now 1781825647345277 us; error 49 us; skew 500 ppm
Jun 18 23:34:07 dist-test-slave-6224 krb5kdc[12151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:07.348402 13114 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:34:07.348886 13114 webserver.cc:460] Webserver started at http://127.0.98.62:46789/ using document root <none> and password file <none>
I20260618 23:34:07.349126 13114 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:07.349205 13114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 18 23:34:07 dist-test-slave-6224 krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260618 23:34:07.427089 13114 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260618 23:34:07.428198 13130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:07.428365 13114 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260618 23:34:07.428462 13114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "05ba038aa32d47a7a34d7a88eaae03c1"
format_stamp: "Formatted at 2026-06-18 23:34:05 on dist-test-slave-6224"
server_key: "98643215cfc2e571ca588c6c5671af07"
server_key_iv: "f455ce5dd1abaf48e70e970fdf094717"
server_key_version: "kuduclusterkey@0"
I20260618 23:34:07.428999 13114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:07.443964 13114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:07.445956 13114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:07.446108 13114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:07.450872 13114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35197
I20260618 23:34:07.451028 13182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35197 every 8 connection(s)
I20260618 23:34:07.451341 13114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:34:07.453080 13183 sys_catalog.cc:263] Verifying existing consensus state
I20260618 23:34:07.453890 13183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1: Bootstrap starting.
I20260618 23:34:07.456493 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 13114
I20260618 23:34:07.456842 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:42879/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37345
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35197
--webserver_port=37241
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:42879/kms with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:34:07.464797 13183 log.cc:824] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:07.466933 13183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:34:07.467392 13183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1: Bootstrap complete.
I20260618 23:34:07.470369 13183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ba038aa32d47a7a34d7a88eaae03c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35197 } }
I20260618 23:34:07.470701 13183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05ba038aa32d47a7a34d7a88eaae03c1, State: Initialized, Role: FOLLOWER
I20260618 23:34:07.470921 13183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ba038aa32d47a7a34d7a88eaae03c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35197 } }
I20260618 23:34:07.471028 13183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:34:07.471087 13183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:34:07.471146 13183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [term 1 FOLLOWER]: Advancing to term 2
I20260618 23:34:07.472540 13183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ba038aa32d47a7a34d7a88eaae03c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35197 } }
I20260618 23:34:07.472996 13183 leader_election.cc:302] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05ba038aa32d47a7a34d7a88eaae03c1; no voters:
I20260618 23:34:07.473340 13183 leader_election.cc:288] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260618 23:34:07.473467 13187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260618 23:34:07.473737 13183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:34:07.473779 13187 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [term 2 LEADER]: Becoming Leader. State: Replica: 05ba038aa32d47a7a34d7a88eaae03c1, State: Running, Role: LEADER
I20260618 23:34:07.473932 13187 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ba038aa32d47a7a34d7a88eaae03c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35197 } }
I20260618 23:34:07.475013 13188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "05ba038aa32d47a7a34d7a88eaae03c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ba038aa32d47a7a34d7a88eaae03c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35197 } } }
I20260618 23:34:07.475138 13188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [sys.catalog]: This master's current role is: LEADER
I20260618 23:34:07.475086 13196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05ba038aa32d47a7a34d7a88eaae03c1. Latest consensus state: current_term: 2 leader_uuid: "05ba038aa32d47a7a34d7a88eaae03c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05ba038aa32d47a7a34d7a88eaae03c1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35197 } } }
I20260618 23:34:07.475445 13196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1 [sys.catalog]: This master's current role is: LEADER
I20260618 23:34:07.475790 13199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:34:07.476002 13183 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260618 23:34:07.476395 13199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:34:07.476557 13199 catalog_manager.cc:1259] Loaded cluster ID: 601fda8234bc4d4586ff8343863289b6
I20260618 23:34:07.476601 13199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:34:07.481825 13199 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:34:07.482252 13199 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1: Loaded TSK: 0
I20260618 23:34:07.482538 13199 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 23:34:07.571200 13185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:07.571480 13185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:07.571519 13185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:07.573463 13185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:07.573532 13185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:07.573583 13185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:34:07.576007 13185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:42879/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37241
--tserver_master_addrs=127.0.98.62:35197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:07.576275 13185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:07.576541 13185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:07.583381 13224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:07.583395 13221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:07.583834 13222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:07.584160 13185 server_base.cc:1034] running on GCE node
I20260618 23:34:07.584405 13185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:07.584695 13185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:07.585865 13185 hybrid_clock.cc:648] HybridClock initialized: now 1781825647585845 us; error 36 us; skew 500 ppm
Jun 18 23:34:07 dist-test-slave-6224 krb5kdc[12151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:07.592144 13185 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:34:07.592760 13185 webserver.cc:460] Webserver started at http://127.0.98.1:37241/ using document root <none> and password file <none>
I20260618 23:34:07.593061 13185 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:07.593154 13185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 18 23:34:07 dist-test-slave-6224 krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260618 23:34:07.615384 13185 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:07.616272 13232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:07.616466 13185 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260618 23:34:07.616546 13185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "0882c9a09d3a408792782116d8943dcc"
format_stamp: "Formatted at 2026-06-18 23:34:06 on dist-test-slave-6224"
server_key: "6d8420e405c380edc35fcf4f27213342"
server_key_iv: "dcf90095eafc090157be67943e633e36"
server_key_version: "kuduclusterkey@0"
I20260618 23:34:07.616981 13185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:07.639539 13185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:07.642087 13185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:07.642258 13185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:07.642485 13185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:34:07.642884 13185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:34:07.642944 13185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:07.642977 13185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:34:07.642997 13185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:07.651257 13185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37345
I20260618 23:34:07.651684 13345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37345 every 8 connection(s)
I20260618 23:34:07.652136 13185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:34:07.653506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 13185
I20260618 23:34:07.653815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:42879/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:32849
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35197
--webserver_port=35637
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:42879/kms with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 23:34:07 dist-test-slave-6224 krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:34:07.671092 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:07.654560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44627 (local address 127.0.98.62:35197)
0618 23:34:07.654813 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:07.654823 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:07.656035 (+ 1212us) server_negotiation.cc:407] Connection header received
0618 23:34:07.656767 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:07.656778 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:07.656858 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:07.657086 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:07.657867 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:07.659361 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:07.661301 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:07.662155 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:07.662320 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:07.666430 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:07.666437 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:07.666437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:07.666454 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:07.668707 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:07.669240 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:07.669243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:07.669244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:07.669362 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:07.669637 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:07.669638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:07.669638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:07.669978 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:07.670075 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:07.670696 (+ 621us) server_negotiation.cc:299] Negotiation successful
0618 23:34:07.670854 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":93,"threads_started":1}
I20260618 23:34:07.671098 13348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:07.653763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35197 (local address 127.0.98.1:44627)
0618 23:34:07.654922 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:07.654970 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:07.656426 (+ 1456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:07.657156 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:07.657160 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:07.657423 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:07.657700 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:07.657705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:07.659578 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:07.659580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:07.661151 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:07.661158 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:07.662547 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:07.662549 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:07.662764 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:07.663471 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:07.663484 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:07.666224 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:07.668812 (+ 2588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:07.668818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:07.668820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:07.669137 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:07.669443 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:07.669444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:07.669444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:07.669568 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:07.670106 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:07.670121 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:07.670396 (+ 275us) client_negotiation.cc:769] Sending connection context
0618 23:34:07.670555 (+ 159us) client_negotiation.cc:240] Negotiation successful
0618 23:34:07.670801 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1115,"thread_start_us":107,"threads_started":1}
I20260618 23:34:07.672715 13346 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35197
I20260618 23:34:07.672891 13346 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:07.673151 13346 heartbeater.cc:511] Master 127.0.98.62:35197 requested a full tablet report, sending...
W20260618 23:34:07.674296 13346 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35197 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260618 23:34:07.674378 13346 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:07.674443 13346 heartbeater.cc:511] Master 127.0.98.62:35197 requested a full tablet report, sending...
I20260618 23:34:07.674908 13346 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:07.674988 13346 heartbeater.cc:511] Master 127.0.98.62:35197 requested a full tablet report, sending...
W20260618 23:34:07.675419 13346 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 23:34:07.793174 13350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:07.793541 13350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:07.793605 13350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:07.795923 13350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:07.796002 13350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:07.796054 13350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:34:07.798847 13350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:42879/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:32849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35637
--tserver_master_addrs=127.0.98.62:35197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:07.799154 13350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:07.799530 13350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:07.806360 13355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:07.807780 13358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:07.808265 13356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:07.808517 13350 server_base.cc:1034] running on GCE node
I20260618 23:34:07.808764 13350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:07.809094 13350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:07.810284 13350 hybrid_clock.cc:648] HybridClock initialized: now 1781825647810244 us; error 61 us; skew 500 ppm
Jun 18 23:34:07 dist-test-slave-6224 krb5kdc[12151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:07.816776 13350 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:34:07.817512 13350 webserver.cc:460] Webserver started at http://127.0.98.2:35637/ using document root <none> and password file <none>
I20260618 23:34:07.817875 13350 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:07.817971 13350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 18 23:34:07 dist-test-slave-6224 krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260618 23:34:07.838896 13350 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:07.848151 13366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:07.848557 13350 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.001s sys 0.000s
I20260618 23:34:07.848737 13350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "1fa78cab745a413c9848b35668d9a903"
format_stamp: "Formatted at 2026-06-18 23:34:06 on dist-test-slave-6224"
server_key: "c4c5bc683059a2048779b94169a143d6"
server_key_iv: "770a689008fa7b16e44a87f1e7e741bf"
server_key_version: "kuduclusterkey@0"
I20260618 23:34:07.849351 13350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:07.867168 13350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:07.869746 13350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:07.869997 13350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:07.870293 13350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:34:07.870735 13350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:34:07.870795 13350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:07.870831 13350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:34:07.870852 13350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:07.879444 13350 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32849
I20260618 23:34:07.880167 13350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:34:07.880862 13479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32849 every 8 connection(s)
I20260618 23:34:07.882686 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 13350
I20260618 23:34:07.883069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:42879/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42513
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35197
--webserver_port=36393
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:42879/kms with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 23:34:07 dist-test-slave-6224 krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:34:07.894358 13482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:07.881203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35197 (local address 127.0.98.2:49765)
0618 23:34:07.881426 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:07.881478 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:07.882261 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:07.882749 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:07.882755 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:07.883069 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:07.883416 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:07.883422 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:07.884576 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:07.884578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:07.886001 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:07.886005 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:07.886645 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:07.886647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:07.886846 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:07.887676 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:07.887687 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:07.889348 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:07.891912 (+ 2564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:07.891921 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:07.891926 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:07.892372 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:07.892759 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:07.892762 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:07.892762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:07.892978 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:07.893367 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:07.893374 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:07.893584 (+ 210us) client_negotiation.cc:769] Sending connection context
0618 23:34:07.893707 (+ 123us) client_negotiation.cc:240] Negotiation successful
0618 23:34:07.894054 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":130,"threads_started":1}
I20260618 23:34:07.895082 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:07.881186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49765 (local address 127.0.98.62:35197)
0618 23:34:07.881261 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:07.881264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:07.881579 (+ 315us) server_negotiation.cc:407] Connection header received
0618 23:34:07.882495 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:07.882497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:07.882516 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:07.882630 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:07.883537 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:07.884408 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:07.886136 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:07.886438 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:07.886534 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:07.889474 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:07.889476 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:07.889477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:07.889486 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:07.891655 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:07.892501 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:07.892504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:07.892504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:07.892630 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:07.893068 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:07.893069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:07.893070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:07.893239 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:07.893316 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:07.894788 (+ 1472us) server_negotiation.cc:299] Negotiation successful
0618 23:34:07.894983 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 23:34:07.896174 13480 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35197
I20260618 23:34:07.896346 13480 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:07.896667 13480 heartbeater.cc:511] Master 127.0.98.62:35197 requested a full tablet report, sending...
W20260618 23:34:07.897571 13480 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35197 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260618 23:34:07.897692 13480 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:07.897809 13480 heartbeater.cc:511] Master 127.0.98.62:35197 requested a full tablet report, sending...
I20260618 23:34:07.898387 13480 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:07.898520 13480 heartbeater.cc:511] Master 127.0.98.62:35197 requested a full tablet report, sending...
W20260618 23:34:07.899026 13480 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 23:34:08.007887 13483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:08.008164 13483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:08.008208 13483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:08.009513 13483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:08.009593 13483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:08.009641 13483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:34:08.011327 13483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:42879/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36393
--tserver_master_addrs=127.0.98.62:35197
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:08.011650 13483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:08.011966 13483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:08.018226 13491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:08.018350 13488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:08.018549 13489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:08.019219 13483 server_base.cc:1034] running on GCE node
I20260618 23:34:08.019598 13483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:08.019948 13483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:08.021116 13483 hybrid_clock.cc:648] HybridClock initialized: now 1781825648021097 us; error 37 us; skew 500 ppm
Jun 18 23:34:08 dist-test-slave-6224 krb5kdc[12151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:08.025254 13483 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:34:08.025902 13483 webserver.cc:460] Webserver started at http://127.0.98.3:36393/ using document root <none> and password file <none>
I20260618 23:34:08.026181 13483 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:08.026269 13483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 18 23:34:08 dist-test-slave-6224 krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260618 23:34:08.056160 13483 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.001s sys 0.000s
I20260618 23:34:08.059217 13500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:08.059432 13483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.000s
I20260618 23:34:08.059537 13483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "33b0dfcf52554b55ba9876bf8a5f9403"
format_stamp: "Formatted at 2026-06-18 23:34:07 on dist-test-slave-6224"
server_key: "604148be007d4a1e40f531cc9e1dd5e2"
server_key_iv: "ea442b5954da657e301592cedef39480"
server_key_version: "kuduclusterkey@0"
I20260618 23:34:08.060184 13483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:08.083861 13483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:08.086675 13483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:08.086861 13483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:08.087174 13483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:34:08.087654 13483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:34:08.087709 13483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:08.087752 13483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:34:08.087781 13483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:08.096092 13483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42513
I20260618 23:34:08.096686 13483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:34:08.100378 13613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42513 every 8 connection(s)
I20260618 23:34:08.105450 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 13483
Jun 18 23:34:08 dist-test-slave-6224 krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:34:08.117498 13349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:08.104488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57677 (local address 127.0.98.62:35197)
0618 23:34:08.104539 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:08.104541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:08.104978 (+ 437us) server_negotiation.cc:407] Connection header received
0618 23:34:08.106295 (+ 1317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:08.106296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:08.106307 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:08.106337 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:08.107226 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:08.108077 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:08.109801 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:08.110119 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:08.110166 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:08.114043 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:08.114045 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:08.114045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:08.114053 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:08.115839 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:08.116362 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:08.116363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:08.116363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:08.116421 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:08.116687 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:08.116688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:08.116688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:08.116830 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:08.116872 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:08.117294 (+ 422us) server_negotiation.cc:299] Negotiation successful
0618 23:34:08.117385 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260618 23:34:08.118083 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:08.104505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35197 (local address 127.0.98.3:57677)
0618 23:34:08.104782 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:08.104839 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:08.106167 (+ 1328us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:08.106541 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:08.106546 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:08.106806 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:08.107092 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:08.107097 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:08.108277 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:08.108279 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:08.109682 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:08.109686 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:08.110331 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:08.110332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:08.110490 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:08.111455 (+ 965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:08.111469 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:08.113924 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:08.115942 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:08.115949 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:08.115952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:08.116274 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:08.116490 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:08.116490 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:08.116491 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:08.116621 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:08.116917 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:08.116922 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:08.117124 (+ 202us) client_negotiation.cc:769] Sending connection context
0618 23:34:08.117605 (+ 481us) client_negotiation.cc:240] Negotiation successful
0618 23:34:08.117807 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":160,"threads_started":1}
I20260618 23:34:08.123652 13614 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35197
I20260618 23:34:08.123864 13614 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:08.124161 13614 heartbeater.cc:511] Master 127.0.98.62:35197 requested a full tablet report, sending...
W20260618 23:34:08.124990 13614 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35197 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260618 23:34:08.125083 13614 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:08.125173 13614 heartbeater.cc:511] Master 127.0.98.62:35197 requested a full tablet report, sending...
I20260618 23:34:08.125556 13618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:08.106110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52250 (local address 127.0.98.62:35197)
0618 23:34:08.107431 (+ 1321us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:08.107433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:08.107440 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:34:08.107458 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:08.107459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:08.107470 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:08.107519 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:08.108297 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:08.109106 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:08.110200 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:08.110500 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:08.110576 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:08.112531 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:08.112533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:08.112533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:08.112540 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:08.114586 (+ 2046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:08.121383 (+ 6797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:08.121384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:08.121385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:08.121472 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:08.124207 (+ 2735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:08.124208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:08.124209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:08.124436 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:08.124640 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:08.125307 (+ 667us) server_negotiation.cc:299] Negotiation successful
0618 23:34:08.125434 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1304,"thread_start_us":42,"threads_started":1}
I20260618 23:34:08.125628 13614 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:08.125705 13614 heartbeater.cc:511] Master 127.0.98.62:35197 requested a full tablet report, sending...
W20260618 23:34:08.126225 13614 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 18 23:34:08 dist-test-slave-6224 krb5kdc[12151](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781825648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:08.675626 13346 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:08.675817 13346 heartbeater.cc:511] Master 127.0.98.62:35197 requested a full tablet report, sending...
I20260618 23:34:08.676393 13147 ts_manager.cc:194] Registered new tserver with Master: 0882c9a09d3a408792782116d8943dcc (127.0.98.1:37345)
I20260618 23:34:08.677270 13147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44627
I20260618 23:34:08.899212 13480 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:08.899407 13480 heartbeater.cc:511] Master 127.0.98.62:35197 requested a full tablet report, sending...
I20260618 23:34:08.899858 13147 ts_manager.cc:194] Registered new tserver with Master: 1fa78cab745a413c9848b35668d9a903 (127.0.98.2:32849)
I20260618 23:34:08.900472 13147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49765
Jun 18 23:34:09 dist-test-slave-6224 krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260618 23:34:09.126432 13614 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:09.126618 13614 heartbeater.cc:511] Master 127.0.98.62:35197 requested a full tablet report, sending...
I20260618 23:34:09.127045 13147 ts_manager.cc:194] Registered new tserver with Master: 33b0dfcf52554b55ba9876bf8a5f9403 (127.0.98.3:42513)
I20260618 23:34:09.127511 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:34:09.127635 13147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57677
I20260618 23:34:09.139170 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:09.128918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52266 (local address 127.0.98.62:35197)
0618 23:34:09.129080 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:09.129083 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:09.129247 (+ 164us) server_negotiation.cc:407] Connection header received
0618 23:34:09.129269 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:09.129270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:09.129281 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:09.129336 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:09.130108 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.130878 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.131955 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.132256 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.132331 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:09.134217 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:09.134219 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:09.134220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:09.134230 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:09.137509 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:09.138011 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:09.138013 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:09.138013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:09.138074 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:09.138260 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:09.138261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:09.138261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:09.138401 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:09.138494 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:09.138969 (+ 475us) server_negotiation.cc:299] Negotiation successful
0618 23:34:09.139056 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
I20260618 23:34:09.141496 13147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:34:09.144165 13147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:34:09.165894 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:09.154318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42513 (local address 127.0.0.1:49792)
0618 23:34:09.155213 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:09.155262 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:09.155327 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:09.155937 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:09.155940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:09.155943 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:34:09.156031 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:09.156033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.158670 (+ 2637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.158671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:09.160428 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:09.160431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.165540 (+ 5109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.165541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:09.165649 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:09.165669 (+ 20us) client_negotiation.cc:769] Sending connection context
0618 23:34:09.165723 (+ 54us) client_negotiation.cc:240] Negotiation successful
0618 23:34:09.165783 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"thread_start_us":88,"threads_started":1}
I20260618 23:34:09.166769 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:09.154779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49792 (local address 127.0.98.3:42513)
0618 23:34:09.155754 (+ 975us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:09.155758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:09.155767 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:34:09.155803 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:09.155805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:09.155831 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:09.156101 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:34:09.156171 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.157587 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.163240 (+ 5653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.164079 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.166481 (+ 2402us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:09.166572 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:09.166604 (+ 32us) server_negotiation.cc:299] Negotiation successful
0618 23:34:09.166665 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":945,"thread_start_us":72,"threads_started":1}
I20260618 23:34:09.170563 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:09.154595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37345 (local address 127.0.0.1:52486)
0618 23:34:09.157823 (+ 3228us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:09.157832 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:09.158299 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:09.158551 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:09.158552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:09.158554 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:34:09.158590 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:09.158591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.164247 (+ 5656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.164248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:09.166369 (+ 2121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:09.166373 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.170333 (+ 3960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.170334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:09.170397 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:09.170402 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:34:09.170437 (+ 35us) client_negotiation.cc:240] Negotiation successful
0618 23:34:09.170481 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3184,"thread_start_us":61,"threads_started":1}
I20260618 23:34:09.170853 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:09.154936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52486 (local address 127.0.98.1:37345)
0618 23:34:09.155512 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:09.155519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:09.157889 (+ 2370us) server_negotiation.cc:407] Connection header received
0618 23:34:09.158389 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:09.158391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:09.158407 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:09.158456 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:34:09.161736 (+ 3280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.163125 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.169072 (+ 5947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.170141 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.170191 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:09.170271 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:09.170726 (+ 455us) server_negotiation.cc:299] Negotiation successful
0618 23:34:09.170773 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":67,"threads_started":1}
I20260618 23:34:09.171175 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:09.154401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32849 (local address 127.0.0.1:41480)
0618 23:34:09.155592 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:09.155602 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:09.155648 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:09.158178 (+ 2530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:09.158179 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:09.158182 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:34:09.158230 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:09.158232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.166954 (+ 8722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.166956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:09.168058 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:09.168061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.170994 (+ 2933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.170995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:09.171052 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:09.171056 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:34:09.171083 (+ 27us) client_negotiation.cc:240] Negotiation successful
0618 23:34:09.171118 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1176,"thread_start_us":104,"threads_started":1}
I20260618 23:34:09.171586 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:09.154488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41480 (local address 127.0.98.2:32849)
0618 23:34:09.157968 (+ 3480us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:09.157973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:09.157981 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:34:09.158015 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:09.158017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:09.158032 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:09.158101 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:34:09.160521 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.161615 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.168173 (+ 6558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.168948 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.171314 (+ 2366us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:09.171416 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:09.171442 (+ 26us) server_negotiation.cc:299] Negotiation successful
0618 23:34:09.171490 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3446,"thread_start_us":69,"threads_started":1}
I20260618 23:34:09.172260 13280 tablet_service.cc:1467] Processing CreateTablet for tablet 84bcfad5459948f0a5c8a4aa59793544 (DEFAULT_TABLE table=test-table [id=61830e0fb77744fa85ffb2814140ccda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:34:09.172821 13280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84bcfad5459948f0a5c8a4aa59793544. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:34:09.179824 13414 tablet_service.cc:1467] Processing CreateTablet for tablet 84bcfad5459948f0a5c8a4aa59793544 (DEFAULT_TABLE table=test-table [id=61830e0fb77744fa85ffb2814140ccda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:34:09.180263 13414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84bcfad5459948f0a5c8a4aa59793544. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:34:09.184047 13545 tablet_service.cc:1467] Processing CreateTablet for tablet 84bcfad5459948f0a5c8a4aa59793544 (DEFAULT_TABLE table=test-table [id=61830e0fb77744fa85ffb2814140ccda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:34:09.184415 13545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84bcfad5459948f0a5c8a4aa59793544. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:34:09.191839 13647 tablet_bootstrap.cc:492] T 84bcfad5459948f0a5c8a4aa59793544 P 0882c9a09d3a408792782116d8943dcc: Bootstrap starting.
I20260618 23:34:09.192983 13647 tablet_bootstrap.cc:654] T 84bcfad5459948f0a5c8a4aa59793544 P 0882c9a09d3a408792782116d8943dcc: Neither blocks nor log segments found. Creating new log.
I20260618 23:34:09.193480 13647 log.cc:824] T 84bcfad5459948f0a5c8a4aa59793544 P 0882c9a09d3a408792782116d8943dcc: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:09.194113 13648 tablet_bootstrap.cc:492] T 84bcfad5459948f0a5c8a4aa59793544 P 33b0dfcf52554b55ba9876bf8a5f9403: Bootstrap starting.
I20260618 23:34:09.195065 13648 tablet_bootstrap.cc:654] T 84bcfad5459948f0a5c8a4aa59793544 P 33b0dfcf52554b55ba9876bf8a5f9403: Neither blocks nor log segments found. Creating new log.
I20260618 23:34:09.195643 13648 log.cc:824] T 84bcfad5459948f0a5c8a4aa59793544 P 33b0dfcf52554b55ba9876bf8a5f9403: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:09.196796 13647 tablet_bootstrap.cc:492] T 84bcfad5459948f0a5c8a4aa59793544 P 0882c9a09d3a408792782116d8943dcc: No bootstrap required, opened a new log
I20260618 23:34:09.196936 13647 ts_tablet_manager.cc:1397] T 84bcfad5459948f0a5c8a4aa59793544 P 0882c9a09d3a408792782116d8943dcc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260618 23:34:09.200525 13647 raft_consensus.cc:348] T 84bcfad5459948f0a5c8a4aa59793544 P 0882c9a09d3a408792782116d8943dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b0dfcf52554b55ba9876bf8a5f9403" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42513 } } peers { permanent_uuid: "0882c9a09d3a408792782116d8943dcc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37345 } } peers { permanent_uuid: "1fa78cab745a413c9848b35668d9a903" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32849 } }
I20260618 23:34:09.200995 13647 raft_consensus.cc:374] T 84bcfad5459948f0a5c8a4aa59793544 P 0882c9a09d3a408792782116d8943dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:34:09.201092 13647 raft_consensus.cc:729] T 84bcfad5459948f0a5c8a4aa59793544 P 0882c9a09d3a408792782116d8943dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0882c9a09d3a408792782116d8943dcc, State: Initialized, Role: FOLLOWER
I20260618 23:34:09.201467 13647 consensus_queue.cc:260] T 84bcfad5459948f0a5c8a4aa59793544 P 0882c9a09d3a408792782116d8943dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b0dfcf52554b55ba9876bf8a5f9403" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42513 } } peers { permanent_uuid: "0882c9a09d3a408792782116d8943dcc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37345 } } peers { permanent_uuid: "1fa78cab745a413c9848b35668d9a903" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32849 } }
I20260618 23:34:09.201968 13647 ts_tablet_manager.cc:1428] T 84bcfad5459948f0a5c8a4aa59793544 P 0882c9a09d3a408792782116d8943dcc: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260618 23:34:09.202162 13346 heartbeater.cc:503] Master 127.0.98.62:35197 was elected leader, sending a full tablet report...
I20260618 23:34:09.204228 13649 tablet_bootstrap.cc:492] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903: Bootstrap starting.
I20260618 23:34:09.204632 13648 tablet_bootstrap.cc:492] T 84bcfad5459948f0a5c8a4aa59793544 P 33b0dfcf52554b55ba9876bf8a5f9403: No bootstrap required, opened a new log
I20260618 23:34:09.204720 13648 ts_tablet_manager.cc:1397] T 84bcfad5459948f0a5c8a4aa59793544 P 33b0dfcf52554b55ba9876bf8a5f9403: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260618 23:34:09.205518 13649 tablet_bootstrap.cc:654] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903: Neither blocks nor log segments found. Creating new log.
I20260618 23:34:09.205824 13649 log.cc:824] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:09.207849 13648 raft_consensus.cc:348] T 84bcfad5459948f0a5c8a4aa59793544 P 33b0dfcf52554b55ba9876bf8a5f9403 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b0dfcf52554b55ba9876bf8a5f9403" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42513 } } peers { permanent_uuid: "0882c9a09d3a408792782116d8943dcc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37345 } } peers { permanent_uuid: "1fa78cab745a413c9848b35668d9a903" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32849 } }
I20260618 23:34:09.208069 13648 raft_consensus.cc:374] T 84bcfad5459948f0a5c8a4aa59793544 P 33b0dfcf52554b55ba9876bf8a5f9403 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:34:09.208106 13648 raft_consensus.cc:729] T 84bcfad5459948f0a5c8a4aa59793544 P 33b0dfcf52554b55ba9876bf8a5f9403 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33b0dfcf52554b55ba9876bf8a5f9403, State: Initialized, Role: FOLLOWER
I20260618 23:34:09.208487 13648 consensus_queue.cc:260] T 84bcfad5459948f0a5c8a4aa59793544 P 33b0dfcf52554b55ba9876bf8a5f9403 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b0dfcf52554b55ba9876bf8a5f9403" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42513 } } peers { permanent_uuid: "0882c9a09d3a408792782116d8943dcc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37345 } } peers { permanent_uuid: "1fa78cab745a413c9848b35668d9a903" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32849 } }
I20260618 23:34:09.208909 13648 ts_tablet_manager.cc:1428] T 84bcfad5459948f0a5c8a4aa59793544 P 33b0dfcf52554b55ba9876bf8a5f9403: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260618 23:34:09.209085 13614 heartbeater.cc:503] Master 127.0.98.62:35197 was elected leader, sending a full tablet report...
I20260618 23:34:09.215372 13649 tablet_bootstrap.cc:492] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903: No bootstrap required, opened a new log
I20260618 23:34:09.215494 13649 ts_tablet_manager.cc:1397] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260618 23:34:09.218374 13649 raft_consensus.cc:348] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b0dfcf52554b55ba9876bf8a5f9403" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42513 } } peers { permanent_uuid: "0882c9a09d3a408792782116d8943dcc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37345 } } peers { permanent_uuid: "1fa78cab745a413c9848b35668d9a903" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32849 } }
I20260618 23:34:09.218592 13649 raft_consensus.cc:374] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:34:09.218636 13649 raft_consensus.cc:729] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa78cab745a413c9848b35668d9a903, State: Initialized, Role: FOLLOWER
I20260618 23:34:09.218832 13649 consensus_queue.cc:260] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b0dfcf52554b55ba9876bf8a5f9403" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42513 } } peers { permanent_uuid: "0882c9a09d3a408792782116d8943dcc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37345 } } peers { permanent_uuid: "1fa78cab745a413c9848b35668d9a903" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32849 } }
I20260618 23:34:09.219165 13649 ts_tablet_manager.cc:1428] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260618 23:34:09.219311 13480 heartbeater.cc:503] Master 127.0.98.62:35197 was elected leader, sending a full tablet report...
W20260618 23:34:09.354913 13615 tablet.cc:2369] T 84bcfad5459948f0a5c8a4aa59793544 P 33b0dfcf52554b55ba9876bf8a5f9403: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:34:09.372692 13655 raft_consensus.cc:482] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:34:09.372861 13655 raft_consensus.cc:504] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b0dfcf52554b55ba9876bf8a5f9403" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42513 } } peers { permanent_uuid: "0882c9a09d3a408792782116d8943dcc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37345 } } peers { permanent_uuid: "1fa78cab745a413c9848b35668d9a903" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32849 } }
I20260618 23:34:09.373245 13655 leader_election.cc:288] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33b0dfcf52554b55ba9876bf8a5f9403 (127.0.98.3:42513), 0882c9a09d3a408792782116d8943dcc (127.0.98.1:37345)
W20260618 23:34:09.382138 13481 tablet.cc:2369] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:34:09.382531 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:09.373915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58943 (local address 127.0.98.1:37345)
0618 23:34:09.373962 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:09.373965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:09.378857 (+ 4892us) server_negotiation.cc:407] Connection header received
0618 23:34:09.378975 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:09.378977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:09.378992 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:09.379021 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:34:09.379278 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.380045 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.381531 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.382125 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.382175 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:09.382187 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:09.382410 (+ 223us) server_negotiation.cc:299] Negotiation successful
0618 23:34:09.382453 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:34:09.382835 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:09.373718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37345 (local address 127.0.98.2:58943)
0618 23:34:09.378743 (+ 5025us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:09.378761 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:09.378895 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:09.379094 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:09.379094 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:09.379096 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:34:09.379193 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:09.379196 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.380165 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.380166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:09.381424 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:09.381427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.382249 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.382250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:09.382320 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:09.382324 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:34:09.382682 (+ 358us) client_negotiation.cc:240] Negotiation successful
0618 23:34:09.382734 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4999,"thread_start_us":52,"threads_started":1}
I20260618 23:34:09.383193 13296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bcfad5459948f0a5c8a4aa59793544" candidate_uuid: "1fa78cab745a413c9848b35668d9a903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0882c9a09d3a408792782116d8943dcc" is_pre_election: true
I20260618 23:34:09.383385 13296 raft_consensus.cc:2454] T 84bcfad5459948f0a5c8a4aa59793544 P 0882c9a09d3a408792782116d8943dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fa78cab745a413c9848b35668d9a903 in term 0.
I20260618 23:34:09.383651 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:09.373498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42513 (local address 127.0.98.2:34705)
0618 23:34:09.375026 (+ 1528us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:09.375045 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:09.375211 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:09.375435 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:09.375436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:09.375438 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:34:09.375578 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:09.375581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.379167 (+ 3586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.379169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:09.382427 (+ 3258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:09.382435 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.383338 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.383339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:09.383422 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:09.383426 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:34:09.383506 (+ 80us) client_negotiation.cc:240] Negotiation successful
0618 23:34:09.383579 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1495,"thread_start_us":112,"threads_started":1}
I20260618 23:34:09.383684 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:09.374027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34705 (local address 127.0.98.3:42513)
0618 23:34:09.374075 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:09.374077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:09.375166 (+ 1089us) server_negotiation.cc:407] Connection header received
0618 23:34:09.375302 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:09.375304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:09.375321 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:09.375353 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:34:09.375658 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.376438 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.382582 (+ 6144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.383194 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.383244 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:09.383257 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:09.383513 (+ 256us) server_negotiation.cc:299] Negotiation successful
0618 23:34:09.383598 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260618 23:34:09.383657 13368 leader_election.cc:302] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0882c9a09d3a408792782116d8943dcc, 1fa78cab745a413c9848b35668d9a903; no voters:
I20260618 23:34:09.384012 13655 raft_consensus.cc:2781] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:34:09.384081 13655 raft_consensus.cc:482] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:34:09.384120 13655 raft_consensus.cc:3037] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:34:09.384181 13557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bcfad5459948f0a5c8a4aa59793544" candidate_uuid: "1fa78cab745a413c9848b35668d9a903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33b0dfcf52554b55ba9876bf8a5f9403" is_pre_election: true
I20260618 23:34:09.384380 13557 raft_consensus.cc:2454] T 84bcfad5459948f0a5c8a4aa59793544 P 33b0dfcf52554b55ba9876bf8a5f9403 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fa78cab745a413c9848b35668d9a903 in term 0.
I20260618 23:34:09.385149 13655 raft_consensus.cc:504] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b0dfcf52554b55ba9876bf8a5f9403" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42513 } } peers { permanent_uuid: "0882c9a09d3a408792782116d8943dcc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37345 } } peers { permanent_uuid: "1fa78cab745a413c9848b35668d9a903" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32849 } }
I20260618 23:34:09.385306 13655 leader_election.cc:288] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903 [CANDIDATE]: Term 1 election: Requested vote from peers 33b0dfcf52554b55ba9876bf8a5f9403 (127.0.98.3:42513), 0882c9a09d3a408792782116d8943dcc (127.0.98.1:37345)
I20260618 23:34:09.385630 13557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bcfad5459948f0a5c8a4aa59793544" candidate_uuid: "1fa78cab745a413c9848b35668d9a903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33b0dfcf52554b55ba9876bf8a5f9403"
I20260618 23:34:09.385749 13557 raft_consensus.cc:3037] T 84bcfad5459948f0a5c8a4aa59793544 P 33b0dfcf52554b55ba9876bf8a5f9403 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:34:09.386678 13557 raft_consensus.cc:2454] T 84bcfad5459948f0a5c8a4aa59793544 P 33b0dfcf52554b55ba9876bf8a5f9403 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fa78cab745a413c9848b35668d9a903 in term 1.
I20260618 23:34:09.386878 13368 leader_election.cc:302] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa78cab745a413c9848b35668d9a903, 33b0dfcf52554b55ba9876bf8a5f9403; no voters:
I20260618 23:34:09.386999 13296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84bcfad5459948f0a5c8a4aa59793544" candidate_uuid: "1fa78cab745a413c9848b35668d9a903" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0882c9a09d3a408792782116d8943dcc"
I20260618 23:34:09.387085 13296 raft_consensus.cc:3037] T 84bcfad5459948f0a5c8a4aa59793544 P 0882c9a09d3a408792782116d8943dcc [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:34:09.388080 13296 raft_consensus.cc:2454] T 84bcfad5459948f0a5c8a4aa59793544 P 0882c9a09d3a408792782116d8943dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fa78cab745a413c9848b35668d9a903 in term 1.
I20260618 23:34:09.388303 13655 raft_consensus.cc:2781] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:34:09.388485 13655 raft_consensus.cc:686] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903 [term 1 LEADER]: Becoming Leader. State: Replica: 1fa78cab745a413c9848b35668d9a903, State: Running, Role: LEADER
I20260618 23:34:09.388593 13655 consensus_queue.cc:237] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b0dfcf52554b55ba9876bf8a5f9403" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42513 } } peers { permanent_uuid: "0882c9a09d3a408792782116d8943dcc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37345 } } peers { permanent_uuid: "1fa78cab745a413c9848b35668d9a903" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32849 } }
I20260618 23:34:09.389468 13147 catalog_manager.cc:5495] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fa78cab745a413c9848b35668d9a903 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1fa78cab745a413c9848b35668d9a903" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b0dfcf52554b55ba9876bf8a5f9403" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42513 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0882c9a09d3a408792782116d8943dcc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fa78cab745a413c9848b35668d9a903" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32849 } health_report { overall_health: HEALTHY } } }
W20260618 23:34:09.407995 13347 tablet.cc:2369] T 84bcfad5459948f0a5c8a4aa59793544 P 0882c9a09d3a408792782116d8943dcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:34:09.450310 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:09.446894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41496 (local address 127.0.98.2:32849)
0618 23:34:09.447007 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:09.447008 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:09.447014 (+ 6us) server_negotiation.cc:407] Connection header received
0618 23:34:09.447049 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:09.447049 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:09.447068 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:09.447151 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:34:09.447295 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.448026 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.449118 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:09.449430 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:09.449506 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:09.449728 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:34:09.449867 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:34:09.449945 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:09.450202 (+ 257us) server_negotiation.cc:299] Negotiation successful
0618 23:34:09.450226 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
Jun 18 23:34:09 dist-test-slave-6224 krb5kdc[12151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260618 23:34:09.464972 13296 raft_consensus.cc:1261] T 84bcfad5459948f0a5c8a4aa59793544 P 0882c9a09d3a408792782116d8943dcc [term 1 FOLLOWER]: Refusing update from remote peer 1fa78cab745a413c9848b35668d9a903: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:34:09.465358 13658 consensus_queue.cc:1035] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0882c9a09d3a408792782116d8943dcc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:34:09.467231 13557 raft_consensus.cc:1261] T 84bcfad5459948f0a5c8a4aa59793544 P 33b0dfcf52554b55ba9876bf8a5f9403 [term 1 FOLLOWER]: Refusing update from remote peer 1fa78cab745a413c9848b35668d9a903: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:34:09.467530 13658 consensus_queue.cc:1035] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33b0dfcf52554b55ba9876bf8a5f9403" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:34:09.473984 13662 mvcc.cc:205] Tried to move back new op lower bound from 7298357860204867584 to 7298357859896410112. Current Snapshot: MvccSnapshot[applied={T|T < 7298357860204867584}]
I20260618 23:34:09.474129 13663 mvcc.cc:205] Tried to move back new op lower bound from 7298357860204867584 to 7298357859896410112. Current Snapshot: MvccSnapshot[applied={T|T < 7298357860204867584}]
I20260618 23:34:09.476655 13665 mvcc.cc:205] Tried to move back new op lower bound from 7298357860204867584 to 7298357859896410112. Current Snapshot: MvccSnapshot[applied={T|T < 7298357860204867584}]
I20260618 23:34:09.479488 13147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:34:09.479637 13147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:34:09.480422 13147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05ba038aa32d47a7a34d7a88eaae03c1: Sending DeleteTablet for 3 replicas of tablet 84bcfad5459948f0a5c8a4aa59793544
I20260618 23:34:09.480840 13545 tablet_service.cc:1514] Processing DeleteTablet for tablet 84bcfad5459948f0a5c8a4aa59793544 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:34:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49792
I20260618 23:34:09.480978 13280 tablet_service.cc:1514] Processing DeleteTablet for tablet 84bcfad5459948f0a5c8a4aa59793544 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:34:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52486
I20260618 23:34:09.481122 13414 tablet_service.cc:1514] Processing DeleteTablet for tablet 84bcfad5459948f0a5c8a4aa59793544 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:34:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41480
I20260618 23:34:09.482014 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 13185
I20260618 23:34:09.490024 13673 tablet_replica.cc:330] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903: stopping tablet replica
I20260618 23:34:09.490247 13673 raft_consensus.cc:2229] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:34:09.490509 13673 raft_consensus.cc:2258] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:34:09.491591 13671 tablet_replica.cc:330] T 84bcfad5459948f0a5c8a4aa59793544 P 33b0dfcf52554b55ba9876bf8a5f9403: stopping tablet replica
I20260618 23:34:09.491819 13671 raft_consensus.cc:2229] T 84bcfad5459948f0a5c8a4aa59793544 P 33b0dfcf52554b55ba9876bf8a5f9403 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:34:09.491976 13671 raft_consensus.cc:2258] T 84bcfad5459948f0a5c8a4aa59793544 P 33b0dfcf52554b55ba9876bf8a5f9403 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:34:09.492698 13673 ts_tablet_manager.cc:1905] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:34:09.495117 13673 ts_tablet_manager.cc:1918] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:34:09.495262 13673 log.cc:1196] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/84bcfad5459948f0a5c8a4aa59793544
I20260618 23:34:09.495637 13673 ts_tablet_manager.cc:1939] T 84bcfad5459948f0a5c8a4aa59793544 P 1fa78cab745a413c9848b35668d9a903: Deleting consensus metadata
I20260618 23:34:09.496275 13131 catalog_manager.cc:4841] TS 1fa78cab745a413c9848b35668d9a903 (127.0.98.2:32849): tablet 84bcfad5459948f0a5c8a4aa59793544 (table test-table [id=61830e0fb77744fa85ffb2814140ccda]) successfully deleted
I20260618 23:34:09.501943 13671 ts_tablet_manager.cc:1905] T 84bcfad5459948f0a5c8a4aa59793544 P 33b0dfcf52554b55ba9876bf8a5f9403: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:34:09.504313 13671 ts_tablet_manager.cc:1918] T 84bcfad5459948f0a5c8a4aa59793544 P 33b0dfcf52554b55ba9876bf8a5f9403: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:34:09.504459 13671 log.cc:1196] T 84bcfad5459948f0a5c8a4aa59793544 P 33b0dfcf52554b55ba9876bf8a5f9403: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/84bcfad5459948f0a5c8a4aa59793544
I20260618 23:34:09.504798 13671 ts_tablet_manager.cc:1939] T 84bcfad5459948f0a5c8a4aa59793544 P 33b0dfcf52554b55ba9876bf8a5f9403: Deleting consensus metadata
I20260618 23:34:09.505451 13132 catalog_manager.cc:4841] TS 33b0dfcf52554b55ba9876bf8a5f9403 (127.0.98.3:42513): tablet 84bcfad5459948f0a5c8a4aa59793544 (table test-table [id=61830e0fb77744fa85ffb2814140ccda]) successfully deleted
W20260618 23:34:09.512001 13132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37345 (error 108)
I20260618 23:34:09.515311 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 13350
I20260618 23:34:09.517957 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:09.517769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37345 (local address 127.0.0.1:52500)
0618 23:34:09.517855 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:09.517904 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37345: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260618 23:34:09.518252 13132 catalog_manager.cc:4584] TS 0882c9a09d3a408792782116d8943dcc (127.0.98.1:37345): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 84bcfad5459948f0a5c8a4aa59793544: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37345: connect: Connection refused (error 111)
I20260618 23:34:09.524521 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 13483
I20260618 23:34:09.533579 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 13114
I20260618 23:34:09.542016 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260618 23:34:09.886854 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260618 23:34:09.886968 392 mini_ranger.cc:66] Stopping Ranger...
I20260618 23:34:10.232141 392 mini_ranger.cc:68] Stopped Ranger
2026-06-18 23:34:10.232 UTC [12181] LOG: received smart shutdown request
2026-06-18 23:34:10.236 UTC [12181] LOG: background worker "logical replication launcher" (PID 12190) exited with exit code 1
2026-06-18 23:34:10.236 UTC [12185] LOG: shutting down
2026-06-18 23:34:10.418 UTC [12181] LOG: database system is shut down
2026-06-18T23:34:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53621 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:34:10 dist-test-slave-6224 krb5kdc[13684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:34:10 dist-test-slave-6224 krb5kdc[13684](info): set up 2 sockets
Jun 18 23:34:10 dist-test-slave-6224 krb5kdc[13684](info): commencing operation
krb5kdc: starting...
W20260618 23:34:11.628113 392 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:34:11 dist-test-slave-6224 krb5kdc[13684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:34:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:34:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:34:11.762240 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43361
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43361,127.0.98.61:45319,127.0.98.60:44583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:34:11.862109 13697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:11.862457 13697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:11.862504 13697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:11.864349 13697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:34:11.864470 13697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:11.864498 13697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:11.864522 13697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:34:11.864552 13697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:34:11.866701 13697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43361,127.0.98.61:45319,127.0.98.60:44583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:11.867053 13697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:11.867347 13697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:11.873008 13703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:11.872931 13702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:11.873389 13697 server_base.cc:1034] running on GCE node
W20260618 23:34:11.872929 13705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:11.873811 13697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:11.874215 13697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:11.875396 13697 hybrid_clock.cc:648] HybridClock initialized: now 1781825651875358 us; error 68 us; skew 500 ppm
Jun 18 23:34:11 dist-test-slave-6224 krb5kdc[13684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:11.878209 13697 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:34:11.878655 13697 webserver.cc:460] Webserver started at http://127.0.98.62:40913/ using document root <none> and password file <none>
I20260618 23:34:11.878882 13697 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:11.878937 13697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:11.879032 13697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:34:11.880170 13697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "8189688135e84371b71ff42ab60dada3"
format_stamp: "Formatted at 2026-06-18 23:34:11 on dist-test-slave-6224"
server_key: "3a6627cc9fe41ae49075a7f199395277"
server_key_iv: "26745c9d64dd3d6dd1988a88f243cda6"
server_key_version: "encryptionkey@0"
I20260618 23:34:11.880488 13697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "8189688135e84371b71ff42ab60dada3"
format_stamp: "Formatted at 2026-06-18 23:34:11 on dist-test-slave-6224"
server_key: "3a6627cc9fe41ae49075a7f199395277"
server_key_iv: "26745c9d64dd3d6dd1988a88f243cda6"
server_key_version: "encryptionkey@0"
I20260618 23:34:11.881950 13697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:11.882784 13712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:11.883055 13697 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:11.883167 13697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "8189688135e84371b71ff42ab60dada3"
format_stamp: "Formatted at 2026-06-18 23:34:11 on dist-test-slave-6224"
server_key: "3a6627cc9fe41ae49075a7f199395277"
server_key_iv: "26745c9d64dd3d6dd1988a88f243cda6"
server_key_version: "encryptionkey@0"
I20260618 23:34:11.883244 13697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:11.897502 13697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:11.899773 13697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:11.899968 13697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:11.904388 13697 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43361
I20260618 23:34:11.904459 13764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43361 every 8 connection(s)
I20260618 23:34:11.904878 13697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:34:11.906039 13765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:34:11.908006 13765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:34:11.908414 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 13697
I20260618 23:34:11.908543 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:34:11.908742 392 external_mini_cluster.cc:1428] Setting key 104c0de6b5ce30ceba5f8ddbb313785d
Jun 18 23:34:11 dist-test-slave-6224 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260618 23:34:11.920648 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:11.908970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39834 (local address 127.0.98.62:43361)
0618 23:34:11.909269 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:11.909274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:11.909370 (+ 96us) server_negotiation.cc:407] Connection header received
0618 23:34:11.910290 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:11.910292 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:11.910315 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:11.910425 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:11.911196 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:11.912229 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:11.913281 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:11.913854 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:11.914004 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:11.916803 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:11.916812 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:11.916812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:11.916822 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:11.918756 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:11.919312 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:11.919315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:11.919316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:11.919410 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:11.919729 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:11.919730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:11.919730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:11.919958 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:11.920039 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:11.920300 (+ 261us) server_negotiation.cc:299] Negotiation successful
0618 23:34:11.920416 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":95,"threads_started":1}
I20260618 23:34:11.920828 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:11.908948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43361 (local address 127.0.0.1:39834)
0618 23:34:11.909164 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:11.909226 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:11.910102 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:11.910464 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:11.910468 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:11.910875 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:11.911076 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:11.911080 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:11.912347 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:11.912348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:11.913174 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:11.913176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:11.913970 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:11.913971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:11.914043 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:11.914779 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:11.914789 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:11.916706 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:11.918863 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:11.918875 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:11.918878 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:11.919227 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:11.919490 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:11.919491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:11.919491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:11.919661 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:11.920040 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:11.920046 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:11.920168 (+ 122us) client_negotiation.cc:769] Sending connection context
0618 23:34:11.920264 (+ 96us) client_negotiation.cc:240] Negotiation successful
0618 23:34:11.920603 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":98,"thread_start_us":96,"threads_started":1}
I20260618 23:34:11.921679 13765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:34:11.922152 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:11.921983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45319 (local address 127.0.0.1:40268)
0618 23:34:11.922050 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:11.922118 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260618 23:34:11.922317 13715 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45319: connect: Connection refused (error 111)
I20260618 23:34:11.922884 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:11.922778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45319 (local address 127.0.0.1:40278)
0618 23:34:11.922827 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:11.922853 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260618 23:34:11.923048 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45319: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45319: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260618 23:34:11.936825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45319
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43361,127.0.98.61:45319,127.0.98.60:44583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:34:11.972347 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } attempt: 1
I20260618 23:34:11.973024 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:11.972867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45319 (local address 127.0.0.1:40282)
0618 23:34:11.972941 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:11.972989 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260618 23:34:11.973389 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:11.973296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45319 (local address 127.0.0.1:40284)
0618 23:34:11.973340 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:11.973372 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260618 23:34:11.973479 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45319: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45319: connect: Connection refused (error 111)
W20260618 23:34:12.023967 13772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:12.024240 13772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:12.024277 13772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:12.025456 13772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:34:12.025534 13772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:12.025559 13772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:12.025581 13772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:34:12.025609 13772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:34:12.027525 13772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43361,127.0.98.61:45319,127.0.98.60:44583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:12.027872 13772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:12.028157 13772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:34:12.033529 13772 server_base.cc:1034] running on GCE node
W20260618 23:34:12.033551 13780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:12.033617 13778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:12.033529 13777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:12.034062 13772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:12.034476 13772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:12.035636 13772 hybrid_clock.cc:648] HybridClock initialized: now 1781825652035602 us; error 72 us; skew 500 ppm
Jun 18 23:34:12 dist-test-slave-6224 krb5kdc[13684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:12.039000 13772 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260618 23:34:12.039661 13772 webserver.cc:460] Webserver started at http://127.0.98.61:43643/ using document root <none> and password file <none>
I20260618 23:34:12.039924 13772 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:12.039996 13772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:12.040158 13772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:34:12.041424 13772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-1/data/instance:
uuid: "5e01c1fa2f794aa395ba1c320cfc2349"
format_stamp: "Formatted at 2026-06-18 23:34:12 on dist-test-slave-6224"
server_key: "623be199d911b95b5ace5bd90754f658"
server_key_iv: "8b6b4659a55e0e15e5afab10a9c29d21"
server_key_version: "encryptionkey@0"
I20260618 23:34:12.041643 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } attempt: 2
I20260618 23:34:12.041889 13772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-1/wal/instance:
uuid: "5e01c1fa2f794aa395ba1c320cfc2349"
format_stamp: "Formatted at 2026-06-18 23:34:12 on dist-test-slave-6224"
server_key: "623be199d911b95b5ace5bd90754f658"
server_key_iv: "8b6b4659a55e0e15e5afab10a9c29d21"
server_key_version: "encryptionkey@0"
I20260618 23:34:12.042142 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.041995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45319 (local address 127.0.0.1:40294)
0618 23:34:12.042068 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:12.042104 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260618 23:34:12.042625 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.042498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45319 (local address 127.0.0.1:40300)
0618 23:34:12.042560 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:12.042592 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260618 23:34:12.042748 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45319: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45319: connect: Connection refused (error 111)
I20260618 23:34:12.043931 13772 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260618 23:34:12.045004 13787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:12.045291 13772 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:12.045411 13772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-1/wal
uuid: "5e01c1fa2f794aa395ba1c320cfc2349"
format_stamp: "Formatted at 2026-06-18 23:34:12 on dist-test-slave-6224"
server_key: "623be199d911b95b5ace5bd90754f658"
server_key_iv: "8b6b4659a55e0e15e5afab10a9c29d21"
server_key_version: "encryptionkey@0"
I20260618 23:34:12.045508 13772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:12.080425 13772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:12.083858 13772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:12.084050 13772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:12.089154 13772 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45319
I20260618 23:34:12.089252 13839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45319 every 8 connection(s)
I20260618 23:34:12.089660 13772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-1/data/info.pb
I20260618 23:34:12.090667 13840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:34:12.092420 13840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:34:12.093230 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 13772
I20260618 23:34:12.093379 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-1/wal/instance
I20260618 23:34:12.093629 392 external_mini_cluster.cc:1428] Setting key 4811cbb3f33b937170e471f32d7edc72
Jun 18 23:34:12 dist-test-slave-6224 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:34:12.103889 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.093249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39840 (local address 127.0.98.62:43361)
0618 23:34:12.093342 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:12.093346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:12.093532 (+ 186us) server_negotiation.cc:407] Connection header received
0618 23:34:12.094053 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:12.094055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:12.094073 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:12.094190 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:12.094885 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.095632 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.096827 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.097163 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.097253 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:12.099762 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:12.099765 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:12.099766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:12.099778 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:12.101983 (+ 2205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.102478 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.102480 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.102480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.102566 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.102992 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.102993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.102994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.103200 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:12.103285 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:12.103627 (+ 342us) server_negotiation.cc:299] Negotiation successful
0618 23:34:12.103742 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 23:34:12.104195 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.093160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43361 (local address 127.0.0.1:39840)
0618 23:34:12.093381 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:12.093427 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:12.093944 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:12.094286 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:12.094304 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:12.094572 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:12.094770 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:12.094774 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.095785 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.095788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:12.096702 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:12.096705 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.097287 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.097288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:12.097454 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:12.098117 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:12.098125 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:12.099643 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:12.102108 (+ 2465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:12.102115 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:12.102117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:12.102386 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:12.102742 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:12.102743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:12.102743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:12.102905 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:12.103281 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:12.103287 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:12.103419 (+ 132us) client_negotiation.cc:769] Sending connection context
0618 23:34:12.103517 (+ 98us) client_negotiation.cc:240] Negotiation successful
0618 23:34:12.103815 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260618 23:34:12.105101 13840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 18 23:34:12 dist-test-slave-6224 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260618 23:34:12.112703 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.105424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45319 (local address 127.0.0.1:40306)
0618 23:34:12.105498 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:12.105507 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:12.105546 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:12.105963 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:12.105964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:12.106116 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:12.106162 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:12.106163 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.107032 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.107033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:12.107669 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:12.107671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.108140 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.108141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:12.108211 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:12.108654 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:12.108659 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:12.109832 (+ 1173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:12.111638 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:12.111639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:12.111639 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:12.111824 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:12.112038 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:12.112039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:12.112039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:12.112092 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:12.112374 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:12.112375 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:12.112412 (+ 37us) client_negotiation.cc:769] Sending connection context
0618 23:34:12.112483 (+ 71us) client_negotiation.cc:240] Negotiation successful
0618 23:34:12.112583 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260618 23:34:12.112793 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.105532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40306 (local address 127.0.98.61:45319)
0618 23:34:12.105818 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:12.105822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:12.105846 (+ 24us) server_negotiation.cc:407] Connection header received
0618 23:34:12.105863 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:12.105865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:12.105876 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:12.105962 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:12.106234 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.106948 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.107745 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.108063 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.108146 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:12.109916 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:12.109924 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:12.109924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:12.109931 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:12.111507 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.111916 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.111919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.111919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.111974 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.112167 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.112167 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.112168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.112303 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:12.112398 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:12.112512 (+ 114us) server_negotiation.cc:299] Negotiation successful
0618 23:34:12.112659 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":258,"thread_start_us":171,"threads_started":1}
I20260618 23:34:12.113418 13840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:34:12.113858 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.113746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44583 (local address 127.0.0.1:57216)
0618 23:34:12.113789 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:12.113832 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260618 23:34:12.114091 13789 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44583: connect: Connection refused (error 111)
I20260618 23:34:12.114702 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.114592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44583 (local address 127.0.0.1:57220)
0618 23:34:12.114646 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:12.114680 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260618 23:34:12.114847 13840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44583: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44583: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260618 23:34:12.126220 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44583
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43361,127.0.98.61:45319,127.0.98.60:44583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:34:12.133935 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } attempt: 3
Jun 18 23:34:12 dist-test-slave-6224 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260618 23:34:12.141912 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.134554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40320 (local address 127.0.98.61:45319)
0618 23:34:12.134643 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:12.134646 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:12.134657 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:34:12.134687 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:12.134688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:12.134707 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:12.134774 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:12.135127 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.135720 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.136420 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.136630 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.136715 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:12.139080 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:12.139082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:12.139082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:12.139089 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:12.140639 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.141108 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.141109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.141110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.141177 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.141408 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.141408 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.141409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.141521 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:12.141578 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:12.141754 (+ 176us) server_negotiation.cc:299] Negotiation successful
0618 23:34:12.141825 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 23:34:12.141978 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.134453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45319 (local address 127.0.0.1:40320)
0618 23:34:12.134528 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:12.134540 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:12.134607 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:12.134780 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:12.134780 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:12.134977 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:12.135045 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:12.135047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.135806 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.135807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:12.136332 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:12.136333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.136715 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.136716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:12.136796 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:12.137439 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:12.137444 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:12.138949 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:12.140739 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:12.140741 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:12.140742 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:12.141002 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:12.141259 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:12.141259 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:12.141260 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:12.141326 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:12.141585 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:12.141586 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:12.141652 (+ 66us) client_negotiation.cc:769] Sending connection context
0618 23:34:12.141728 (+ 76us) client_negotiation.cc:240] Negotiation successful
0618 23:34:12.141854 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260618 23:34:12.142627 13765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:34:12.143038 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.142911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44583 (local address 127.0.0.1:57230)
0618 23:34:12.142976 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:12.143015 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260618 23:34:12.143504 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.143378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44583 (local address 127.0.0.1:57240)
0618 23:34:12.143435 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:12.143472 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260618 23:34:12.143637 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44583: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44583: connect: Connection refused (error 111)
I20260618 23:34:12.164088 13840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } attempt: 1
I20260618 23:34:12.164595 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.164467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44583 (local address 127.0.0.1:57256)
0618 23:34:12.164533 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:12.164569 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260618 23:34:12.164958 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.164847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44583 (local address 127.0.0.1:57262)
0618 23:34:12.164888 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:12.164912 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260618 23:34:12.165045 13840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44583: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44583: connect: Connection refused (error 111)
I20260618 23:34:12.174811 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } attempt: 1
I20260618 23:34:12.175437 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.175234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44583 (local address 127.0.0.1:57274)
0618 23:34:12.175316 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:12.175390 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260618 23:34:12.175871 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.175735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44583 (local address 127.0.0.1:57278)
0618 23:34:12.175795 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:12.175833 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260618 23:34:12.176035 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44583: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44583: connect: Connection refused (error 111)
W20260618 23:34:12.226080 13847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:12.226387 13847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:12.226418 13847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:12.227998 13847 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:34:12.228063 13847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:12.228081 13847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:12.228096 13847 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:34:12.228111 13847 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:34:12.230154 13847 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43361,127.0.98.61:45319,127.0.98.60:44583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:12.230506 13847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:12.230815 13847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:34:12.233180 13840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } attempt: 2
I20260618 23:34:12.233644 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.233525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44583 (local address 127.0.0.1:57284)
0618 23:34:12.233581 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:12.233618 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260618 23:34:12.234166 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.234067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44583 (local address 127.0.0.1:57292)
0618 23:34:12.234121 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:12.234151 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260618 23:34:12.234359 13840 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44583: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44583: connect: Connection refused (error 111)
W20260618 23:34:12.237102 13853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:12.237115 13855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:12.237071 13852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:12.237320 13847 server_base.cc:1034] running on GCE node
I20260618 23:34:12.237673 13847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:12.238076 13847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:12.239249 13847 hybrid_clock.cc:648] HybridClock initialized: now 1781825652239225 us; error 47 us; skew 500 ppm
Jun 18 23:34:12 dist-test-slave-6224 krb5kdc[13684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:12.242103 13847 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260618 23:34:12.242647 13847 webserver.cc:460] Webserver started at http://127.0.98.60:35207/ using document root <none> and password file <none>
I20260618 23:34:12.242894 13847 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:12.242961 13847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:12.243111 13847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:34:12.244309 13847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-2/data/instance:
uuid: "6503f6b3716d43e09be0029551e83d3d"
format_stamp: "Formatted at 2026-06-18 23:34:12 on dist-test-slave-6224"
server_key: "7379c4b74e426b120d08d2754634d35c"
server_key_iv: "3817f4895edbbfdeced6027664b52d61"
server_key_version: "encryptionkey@0"
I20260618 23:34:12.244700 13847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-2/wal/instance:
uuid: "6503f6b3716d43e09be0029551e83d3d"
format_stamp: "Formatted at 2026-06-18 23:34:12 on dist-test-slave-6224"
server_key: "7379c4b74e426b120d08d2754634d35c"
server_key_iv: "3817f4895edbbfdeced6027664b52d61"
server_key_version: "encryptionkey@0"
I20260618 23:34:12.246258 13847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260618 23:34:12.247138 13862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:12.247409 13847 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:34:12.247527 13847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-2/wal
uuid: "6503f6b3716d43e09be0029551e83d3d"
format_stamp: "Formatted at 2026-06-18 23:34:12 on dist-test-slave-6224"
server_key: "7379c4b74e426b120d08d2754634d35c"
server_key_iv: "3817f4895edbbfdeced6027664b52d61"
server_key_version: "encryptionkey@0"
I20260618 23:34:12.247671 13847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:12.251196 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } attempt: 2
I20260618 23:34:12.251796 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.251641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44583 (local address 127.0.0.1:57308)
0618 23:34:12.251708 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:12.251763 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260618 23:34:12.252213 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.252114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44583 (local address 127.0.0.1:57314)
0618 23:34:12.252165 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:12.252194 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44583: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260618 23:34:12.252353 13765 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44583: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44583: connect: Connection refused (error 111)
I20260618 23:34:12.255509 13847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:12.257568 13847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:12.257725 13847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:12.262199 13847 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44583
I20260618 23:34:12.262275 13914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44583 every 8 connection(s)
I20260618 23:34:12.262635 13847 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-2/data/info.pb
I20260618 23:34:12.264012 13915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:34:12.266113 13915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:34:12.272840 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 13847
I20260618 23:34:12.272989 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825489608381-392-0/minicluster-data/master-2/wal/instance
Jun 18 23:34:12 dist-test-slave-6224 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:34:12.273236 392 external_mini_cluster.cc:1428] Setting key 5953ee9d646841382722f85f6c1ef976
I20260618 23:34:12.277086 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.267054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39844 (local address 127.0.98.62:43361)
0618 23:34:12.267146 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:12.267149 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:12.267440 (+ 291us) server_negotiation.cc:407] Connection header received
0618 23:34:12.268198 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:12.268199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:12.268217 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:12.268291 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:12.269019 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.269544 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.270691 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.270886 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.270966 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:12.273939 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:12.273942 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:12.273942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:12.273951 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:12.275533 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.276044 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.276045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.276046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.276109 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.276362 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.276363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.276364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.276496 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:12.276577 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:12.276892 (+ 315us) server_negotiation.cc:299] Negotiation successful
0618 23:34:12.276979 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 23:34:12.277323 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.267019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43361 (local address 127.0.0.1:39844)
0618 23:34:12.267258 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:12.267327 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:12.268063 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:12.268394 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:12.268398 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:12.268686 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:12.268917 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:12.268921 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.269690 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.269691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:12.270584 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:12.270588 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.270974 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.270975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:12.271126 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:12.271791 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:12.271801 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:12.273805 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:12.275677 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:12.275685 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:12.275688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:12.275952 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:12.276173 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:12.276173 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:12.276174 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:12.276295 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:12.276566 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:12.276573 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:12.276738 (+ 165us) client_negotiation.cc:769] Sending connection context
0618 23:34:12.276855 (+ 117us) client_negotiation.cc:240] Negotiation successful
0618 23:34:12.277045 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":122,"threads_started":1}
I20260618 23:34:12.278010 13915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 23:34:12 dist-test-slave-6224 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 18 23:34:12 dist-test-slave-6224 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260618 23:34:12.283006 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.273697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57330 (local address 127.0.98.60:44583)
0618 23:34:12.273905 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:12.273911 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:12.273936 (+ 25us) server_negotiation.cc:407] Connection header received
0618 23:34:12.274013 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:12.274014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:12.274028 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:12.274086 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:12.274903 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.275774 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.276538 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.276889 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.276970 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:12.279706 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:12.279715 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:12.279716 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:12.279722 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:12.281602 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.282039 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.282042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.282043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.282103 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.282343 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.282343 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.282343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.282476 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:12.282592 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:12.282747 (+ 155us) server_negotiation.cc:299] Negotiation successful
0618 23:34:12.282844 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":125,"threads_started":1}
I20260618 23:34:12.286268 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.278431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40328 (local address 127.0.98.61:45319)
0618 23:34:12.278525 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:12.278529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:12.278570 (+ 41us) server_negotiation.cc:407] Connection header received
0618 23:34:12.278634 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:12.278636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:12.278649 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:12.278720 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:12.279016 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.279714 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.280899 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.281129 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.281198 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:12.283385 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:12.283387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:12.283387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:12.283395 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:12.284957 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.285374 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.285376 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.285376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.285437 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.285652 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.285653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.285653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.285793 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:12.285865 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:12.286137 (+ 272us) server_negotiation.cc:299] Negotiation successful
0618 23:34:12.286203 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 23:34:12.286355 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.278338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45319 (local address 127.0.0.1:40328)
0618 23:34:12.278506 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:12.278521 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:12.278578 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:12.278739 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:12.278740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:12.278888 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:12.278937 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:12.278939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.279797 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.279798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:12.280570 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:12.280572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.281218 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.281219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:12.281299 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:12.281685 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:12.281690 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:12.283274 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:12.285047 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:12.285048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:12.285048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:12.285286 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:12.285523 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:12.285523 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:12.285523 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:12.285591 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:12.285902 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:12.285903 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:12.286001 (+ 98us) client_negotiation.cc:769] Sending connection context
0618 23:34:12.286096 (+ 95us) client_negotiation.cc:240] Negotiation successful
0618 23:34:12.286234 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260618 23:34:12.286823 13915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 23:34:12 dist-test-slave-6224 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 18 23:34:12 dist-test-slave-6224 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260618 23:34:12.293190 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.284360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39848 (local address 127.0.98.62:43361)
0618 23:34:12.284437 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:12.284440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:12.284452 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:34:12.284532 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:12.284533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:12.284545 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:12.284612 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:12.285185 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.285872 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.287085 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.287364 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.287442 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:12.290348 (+ 2906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:12.290350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:12.290350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:12.290356 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:12.291956 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.292383 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.292383 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.292384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.292435 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.292652 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.292662 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.292662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.292799 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:12.292947 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:12.293060 (+ 113us) server_negotiation.cc:299] Negotiation successful
0618 23:34:12.293123 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 23:34:12.294693 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.287183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44583 (local address 127.0.0.1:57344)
0618 23:34:12.287242 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:12.287255 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:12.287301 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:12.287454 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:12.287454 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:12.287662 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:12.287714 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:12.287717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.288483 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.288484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:12.289328 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:12.289330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.289815 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.289816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:12.289892 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:12.290254 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:12.290260 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:12.291638 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:12.293551 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:12.293552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:12.293553 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:12.293784 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:12.294018 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:12.294019 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:12.294019 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:12.294084 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:12.294349 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:12.294350 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:12.294389 (+ 39us) client_negotiation.cc:769] Sending connection context
0618 23:34:12.294468 (+ 79us) client_negotiation.cc:240] Negotiation successful
0618 23:34:12.294580 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260618 23:34:12.294749 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.287247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57344 (local address 127.0.98.60:44583)
0618 23:34:12.287296 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:12.287298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:12.287307 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:34:12.287370 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:12.287371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:12.287383 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:12.287438 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:12.287800 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.288389 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.289411 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.289728 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.289822 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:12.291752 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:12.291754 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:12.291754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:12.291761 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:12.293450 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.293870 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.293871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.293871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.293927 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.294157 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.294158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.294159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.294273 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:12.294344 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:12.294491 (+ 147us) server_negotiation.cc:299] Negotiation successful
0618 23:34:12.294633 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":29}
I20260618 23:34:12.296296 13915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d: Bootstrap starting.
I20260618 23:34:12.297189 13915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d: Neither blocks nor log segments found. Creating new log.
I20260618 23:34:12.297508 13915 log.cc:824] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:12.298393 13915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d: No bootstrap required, opened a new log
Jun 18 23:34:12 dist-test-slave-6224 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825651, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260618 23:34:12.300237 13915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189688135e84371b71ff42ab60dada3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } } peers { permanent_uuid: "5e01c1fa2f794aa395ba1c320cfc2349" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } } peers { permanent_uuid: "6503f6b3716d43e09be0029551e83d3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } }
I20260618 23:34:12.300441 13915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:34:12.300485 13915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6503f6b3716d43e09be0029551e83d3d, State: Initialized, Role: FOLLOWER
I20260618 23:34:12.300666 13915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189688135e84371b71ff42ab60dada3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } } peers { permanent_uuid: "5e01c1fa2f794aa395ba1c320cfc2349" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } } peers { permanent_uuid: "6503f6b3716d43e09be0029551e83d3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } }
I20260618 23:34:12.301002 13915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:34:12.301316 13921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189688135e84371b71ff42ab60dada3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } } peers { permanent_uuid: "5e01c1fa2f794aa395ba1c320cfc2349" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } } peers { permanent_uuid: "6503f6b3716d43e09be0029551e83d3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } } }
I20260618 23:34:12.301501 13921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [sys.catalog]: This master's current role is: FOLLOWER
W20260618 23:34:12.303095 13932 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:34:12.303227 13932 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:34:12.303439 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.294058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40342 (local address 127.0.98.61:45319)
0618 23:34:12.294121 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:12.294123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:12.294170 (+ 47us) server_negotiation.cc:407] Connection header received
0618 23:34:12.294253 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:12.294254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:12.294264 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:12.294311 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:12.295296 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.295861 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.296601 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.296803 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.296874 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:12.300036 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:12.300037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:12.300038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:12.300047 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:12.301889 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.302474 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.302475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.302476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.302568 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.302853 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.302854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.302855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.302983 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:12.303190 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:12.303226 (+ 36us) server_negotiation.cc:299] Negotiation successful
0618 23:34:12.303328 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 23:34:12.307054 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:34:12.303474 13847 master_runner.cc:186] Error getting master registration for 127.0.98.62:43361: OK, instance_id { permanent_uuid: "8189688135e84371b71ff42ab60dada3" instance_seqno: 1781825651903054 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 23:34:12.307457 13847 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 23:34:12.317556 13844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.308884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40352 (local address 127.0.98.61:45319)
0618 23:34:12.308937 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:12.308940 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:12.309495 (+ 555us) server_negotiation.cc:407] Connection header received
0618 23:34:12.309599 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:12.309600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:12.309612 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:12.309643 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:12.310401 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.311139 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.312154 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.312442 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.312491 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:12.313971 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:12.313972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:12.313972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:12.313981 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:12.315817 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.316580 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.316581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.316582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.316642 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.316885 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.316886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.316887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.317035 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:12.317108 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:12.317332 (+ 224us) server_negotiation.cc:299] Negotiation successful
0618 23:34:12.317442 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260618 23:34:12.318024 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.308538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39864 (local address 127.0.98.62:43361)
0618 23:34:12.308584 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:12.308586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:12.308866 (+ 280us) server_negotiation.cc:407] Connection header received
0618 23:34:12.308965 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:12.308966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:12.308976 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:12.309009 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:12.309871 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.310639 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.311730 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.312017 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.312056 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:12.313970 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:12.313972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:12.313972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:12.313981 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:12.316645 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.317073 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.317074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.317074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.317129 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.317335 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.317336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.317337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.317474 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:12.317567 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:12.317855 (+ 288us) server_negotiation.cc:299] Negotiation successful
0618 23:34:12.317935 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:34:12.318676 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.308671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57346 (local address 127.0.98.60:44583)
0618 23:34:12.308712 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:12.308714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:12.308852 (+ 138us) server_negotiation.cc:407] Connection header received
0618 23:34:12.308965 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:12.308966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:12.308976 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:12.309009 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:12.310047 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.310844 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.312020 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.312343 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.313022 (+ 679us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:12.314705 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:12.314706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:12.314707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:12.314715 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:12.317192 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.317659 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.317661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.317661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.317720 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.317943 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.317944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.317944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.318149 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:12.318254 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:12.318475 (+ 221us) server_negotiation.cc:299] Negotiation successful
0618 23:34:12.318568 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260618 23:34:12.325510 13840 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } attempt: 3
Jun 18 23:34:12 dist-test-slave-6224 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260618 23:34:12.333344 13842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.325826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44583 (local address 127.0.0.1:57360)
0618 23:34:12.325912 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:12.325931 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:12.326005 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:12.326169 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:12.326170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:12.326394 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:12.326466 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:12.326468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.327169 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.327171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:12.328085 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:12.328088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.328431 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.328433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:12.328495 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:12.329059 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:12.329065 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:12.330578 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:12.332326 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:12.332327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:12.332328 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:12.332501 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:12.332691 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:12.332692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:12.332692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:12.332748 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:12.333015 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:12.333016 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:12.333066 (+ 50us) client_negotiation.cc:769] Sending connection context
0618 23:34:12.333148 (+ 82us) client_negotiation.cc:240] Negotiation successful
0618 23:34:12.333248 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260618 23:34:12.333344 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.325929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.0.98.60:44583)
0618 23:34:12.325984 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:12.325986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:12.325996 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:34:12.326079 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:12.326079 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:12.326088 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:12.326143 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:12.326582 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.327076 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.328171 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.328349 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.328410 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:12.330665 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:12.330666 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:12.330667 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:12.330674 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:12.332243 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.332571 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.332572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.332573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.332629 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.332813 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.332814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.332814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.332926 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:12.333024 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:12.333172 (+ 148us) server_negotiation.cc:299] Negotiation successful
0618 23:34:12.333248 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260618 23:34:12.334843 13840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349: Bootstrap starting.
I20260618 23:34:12.335785 13840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349: Neither blocks nor log segments found. Creating new log.
I20260618 23:34:12.336099 13840 log.cc:824] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:12.336864 13840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349: No bootstrap required, opened a new log
I20260618 23:34:12.338295 13840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189688135e84371b71ff42ab60dada3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } } peers { permanent_uuid: "5e01c1fa2f794aa395ba1c320cfc2349" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } } peers { permanent_uuid: "6503f6b3716d43e09be0029551e83d3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } }
I20260618 23:34:12.338406 13840 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:34:12.338429 13840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e01c1fa2f794aa395ba1c320cfc2349, State: Initialized, Role: FOLLOWER
I20260618 23:34:12.338565 13840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189688135e84371b71ff42ab60dada3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } } peers { permanent_uuid: "5e01c1fa2f794aa395ba1c320cfc2349" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } } peers { permanent_uuid: "6503f6b3716d43e09be0029551e83d3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } }
I20260618 23:34:12.338820 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189688135e84371b71ff42ab60dada3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } } peers { permanent_uuid: "5e01c1fa2f794aa395ba1c320cfc2349" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } } peers { permanent_uuid: "6503f6b3716d43e09be0029551e83d3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } } }
I20260618 23:34:12.338898 13840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:34:12.338927 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349 [sys.catalog]: This master's current role is: FOLLOWER
W20260618 23:34:12.340736 13953 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:34:12.340838 13953 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:34:12.341182 13772 master_runner.cc:186] Error getting master registration for 127.0.98.62:43361: OK, instance_id { permanent_uuid: "8189688135e84371b71ff42ab60dada3" instance_seqno: 1781825651903054 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 23:34:12.344449 13772 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 23:34:12.346153 13921 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:34:12.346297 13921 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189688135e84371b71ff42ab60dada3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } } peers { permanent_uuid: "5e01c1fa2f794aa395ba1c320cfc2349" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } } peers { permanent_uuid: "6503f6b3716d43e09be0029551e83d3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } }
I20260618 23:34:12.346621 13921 leader_election.cc:288] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8189688135e84371b71ff42ab60dada3 (127.0.98.62:43361), 5e01c1fa2f794aa395ba1c320cfc2349 (127.0.98.61:45319)
I20260618 23:34:12.346982 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6503f6b3716d43e09be0029551e83d3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e01c1fa2f794aa395ba1c320cfc2349" is_pre_election: true
I20260618 23:34:12.347136 13814 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6503f6b3716d43e09be0029551e83d3d in term 0.
I20260618 23:34:12.347407 13865 leader_election.cc:302] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e01c1fa2f794aa395ba1c320cfc2349, 6503f6b3716d43e09be0029551e83d3d; no voters:
I20260618 23:34:12.347652 13921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:34:12.347738 13921 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:34:12.347771 13921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:34:12.348897 13921 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189688135e84371b71ff42ab60dada3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } } peers { permanent_uuid: "5e01c1fa2f794aa395ba1c320cfc2349" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } } peers { permanent_uuid: "6503f6b3716d43e09be0029551e83d3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } }
I20260618 23:34:12.349062 13921 leader_election.cc:288] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [CANDIDATE]: Term 1 election: Requested vote from peers 8189688135e84371b71ff42ab60dada3 (127.0.98.62:43361), 5e01c1fa2f794aa395ba1c320cfc2349 (127.0.98.61:45319)
I20260618 23:34:12.349354 13814 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6503f6b3716d43e09be0029551e83d3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e01c1fa2f794aa395ba1c320cfc2349"
I20260618 23:34:12.349453 13814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:34:12.350505 13814 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6503f6b3716d43e09be0029551e83d3d in term 1.
I20260618 23:34:12.350706 13865 leader_election.cc:302] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e01c1fa2f794aa395ba1c320cfc2349, 6503f6b3716d43e09be0029551e83d3d; no voters:
I20260618 23:34:12.350950 13921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:34:12.351159 13921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [term 1 LEADER]: Becoming Leader. State: Replica: 6503f6b3716d43e09be0029551e83d3d, State: Running, Role: LEADER
I20260618 23:34:12.351282 13921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189688135e84371b71ff42ab60dada3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } } peers { permanent_uuid: "5e01c1fa2f794aa395ba1c320cfc2349" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } } peers { permanent_uuid: "6503f6b3716d43e09be0029551e83d3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } }
I20260618 23:34:12.351845 13921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6503f6b3716d43e09be0029551e83d3d. Latest consensus state: current_term: 1 leader_uuid: "6503f6b3716d43e09be0029551e83d3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189688135e84371b71ff42ab60dada3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } } peers { permanent_uuid: "5e01c1fa2f794aa395ba1c320cfc2349" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } } peers { permanent_uuid: "6503f6b3716d43e09be0029551e83d3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } } }
I20260618 23:34:12.351930 13921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [sys.catalog]: This master's current role is: LEADER
I20260618 23:34:12.352082 13765 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } attempt: 3
I20260618 23:34:12.355751 13957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:34:12.356032 13957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 18 23:34:12 dist-test-slave-6224 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260618 23:34:12.358633 13814 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349 [term 1 FOLLOWER]: Refusing update from remote peer 6503f6b3716d43e09be0029551e83d3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:34:12.359031 13955 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e01c1fa2f794aa395ba1c320cfc2349" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:34:12.361946 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6503f6b3716d43e09be0029551e83d3d. Latest consensus state: current_term: 1 leader_uuid: "6503f6b3716d43e09be0029551e83d3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189688135e84371b71ff42ab60dada3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } } peers { permanent_uuid: "5e01c1fa2f794aa395ba1c320cfc2349" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } } peers { permanent_uuid: "6503f6b3716d43e09be0029551e83d3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } } }
I20260618 23:34:12.362087 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:34:12.362315 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.352453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44583 (local address 127.0.0.1:53032)
0618 23:34:12.352567 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:12.352579 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:12.352622 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:12.352863 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:12.352864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:12.353084 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:12.353147 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:12.353149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.354636 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.354637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:12.355475 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:12.355478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.356620 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.356621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:12.356691 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:12.357298 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:12.357304 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:12.358956 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:12.361086 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:12.361087 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:12.361088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:12.361313 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:12.361573 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:12.361574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:12.361574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:12.361646 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:12.361972 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:12.361973 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:12.362034 (+ 61us) client_negotiation.cc:769] Sending connection context
0618 23:34:12.362105 (+ 71us) client_negotiation.cc:240] Negotiation successful
0618 23:34:12.362206 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260618 23:34:12.362923 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:12.352591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53032 (local address 127.0.98.60:44583)
0618 23:34:12.352705 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:12.352707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:12.352715 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:34:12.352739 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:12.352740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:12.352751 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:12.352797 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:12.353268 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.354053 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.356172 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:12.356481 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:12.356544 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:12.359172 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:12.359173 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:12.359173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:12.359200 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:12.360972 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.361407 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.361408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.361408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.361491 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:12.361723 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:12.361724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:12.361724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:12.361867 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:12.361925 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:12.362753 (+ 828us) server_negotiation.cc:299] Negotiation successful
0618 23:34:12.362834 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260618 23:34:12.364364 13921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6503f6b3716d43e09be0029551e83d3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189688135e84371b71ff42ab60dada3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } } peers { permanent_uuid: "5e01c1fa2f794aa395ba1c320cfc2349" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } } peers { permanent_uuid: "6503f6b3716d43e09be0029551e83d3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } } }
I20260618 23:34:12.364487 13921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [sys.catalog]: This master's current role is: LEADER
I20260618 23:34:12.364661 13765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3: Bootstrap starting.
I20260618 23:34:12.365192 13942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6503f6b3716d43e09be0029551e83d3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189688135e84371b71ff42ab60dada3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } } peers { permanent_uuid: "5e01c1fa2f794aa395ba1c320cfc2349" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } } peers { permanent_uuid: "6503f6b3716d43e09be0029551e83d3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } } }
I20260618 23:34:12.365306 13942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:34:12.365433 13765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3: Neither blocks nor log segments found. Creating new log.
I20260618 23:34:12.365752 13765 log.cc:824] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:12.365773 13957 catalog_manager.cc:1347] Generated new cluster ID: 00b685c8f31e4ea0a0894f8bb3d168b2
I20260618 23:34:12.365828 13957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:34:12.372629 13765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3: No bootstrap required, opened a new log
I20260618 23:34:12.374666 13765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189688135e84371b71ff42ab60dada3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } } peers { permanent_uuid: "5e01c1fa2f794aa395ba1c320cfc2349" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } } peers { permanent_uuid: "6503f6b3716d43e09be0029551e83d3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } }
I20260618 23:34:12.374804 13765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:34:12.374835 13765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8189688135e84371b71ff42ab60dada3, State: Initialized, Role: FOLLOWER
I20260618 23:34:12.374884 13957 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:34:12.374956 13765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189688135e84371b71ff42ab60dada3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } } peers { permanent_uuid: "5e01c1fa2f794aa395ba1c320cfc2349" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } } peers { permanent_uuid: "6503f6b3716d43e09be0029551e83d3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } }
I20260618 23:34:12.375268 13765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:34:12.375680 13957 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:34:12.376242 13737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:34:12.377182 13737 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3 [term 1 FOLLOWER]: Refusing update from remote peer 6503f6b3716d43e09be0029551e83d3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:34:12.377521 13921 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8189688135e84371b71ff42ab60dada3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:34:12.378357 13738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6503f6b3716d43e09be0029551e83d3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8189688135e84371b71ff42ab60dada3"
I20260618 23:34:12.379155 13738 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6503f6b3716d43e09be0029551e83d3d in term 1.
I20260618 23:34:12.379711 13739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6503f6b3716d43e09be0029551e83d3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8189688135e84371b71ff42ab60dada3" is_pre_election: true
I20260618 23:34:12.379792 13739 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 6503f6b3716d43e09be0029551e83d3d in term 1. Re-sending same reply.
I20260618 23:34:12.383625 13965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6503f6b3716d43e09be0029551e83d3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189688135e84371b71ff42ab60dada3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } } peers { permanent_uuid: "5e01c1fa2f794aa395ba1c320cfc2349" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } } peers { permanent_uuid: "6503f6b3716d43e09be0029551e83d3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } } }
I20260618 23:34:12.383734 13965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:34:12.383788 13965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6503f6b3716d43e09be0029551e83d3d. Latest consensus state: current_term: 1 leader_uuid: "6503f6b3716d43e09be0029551e83d3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189688135e84371b71ff42ab60dada3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } } peers { permanent_uuid: "5e01c1fa2f794aa395ba1c320cfc2349" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } } peers { permanent_uuid: "6503f6b3716d43e09be0029551e83d3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } } }
I20260618 23:34:12.383842 13965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:34:12.383874 13965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6503f6b3716d43e09be0029551e83d3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189688135e84371b71ff42ab60dada3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } } peers { permanent_uuid: "5e01c1fa2f794aa395ba1c320cfc2349" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } } peers { permanent_uuid: "6503f6b3716d43e09be0029551e83d3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } } }
I20260618 23:34:12.383913 13965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3 [sys.catalog]: This master's current role is: FOLLOWER
W20260618 23:34:12.385350 13980 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:34:12.385495 13980 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:34:12.386428 13955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6503f6b3716d43e09be0029551e83d3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8189688135e84371b71ff42ab60dada3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43361 } } peers { permanent_uuid: "5e01c1fa2f794aa395ba1c320cfc2349" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45319 } } peers { permanent_uuid: "6503f6b3716d43e09be0029551e83d3d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44583 } } }
I20260618 23:34:12.386577 13955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d [sys.catalog]: This master's current role is: LEADER
I20260618 23:34:12.386445 13957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6503f6b3716d43e09be0029551e83d3d: Generated new TSK 0
I20260618 23:34:12.387812 13957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:34:12.388655 13972 mvcc.cc:205] Tried to move back new op lower bound from 7298357872117592064 to 7298357872031608832. Current Snapshot: MvccSnapshot[applied={T|T < 7298357872057921536}]
I20260618 23:34:13.341401 13953 catalog_manager.cc:1259] Loaded cluster ID: 00b685c8f31e4ea0a0894f8bb3d168b2
I20260618 23:34:13.341506 13953 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349: loading cluster ID for follower catalog manager: success
I20260618 23:34:13.342464 13953 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349: acquiring CA information for follower catalog manager: success
I20260618 23:34:13.342841 13953 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5e01c1fa2f794aa395ba1c320cfc2349: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 23:34:13.386090 13980 catalog_manager.cc:1259] Loaded cluster ID: 00b685c8f31e4ea0a0894f8bb3d168b2
I20260618 23:34:13.386219 13980 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3: loading cluster ID for follower catalog manager: success
I20260618 23:34:13.387640 13980 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3: acquiring CA information for follower catalog manager: success
I20260618 23:34:13.388058 13980 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8189688135e84371b71ff42ab60dada3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 23:34:13.454572 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 13697
I20260618 23:34:13.462302 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 13772
I20260618 23:34:13.468495 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 13847
2026-06-18T23:34:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2935 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:34:13 dist-test-slave-6224 krb5kdc[13991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:34:13 dist-test-slave-6224 krb5kdc[13991](info): set up 2 sockets
Jun 18 23:34:13 dist-test-slave-6224 krb5kdc[13991](info): commencing operation
krb5kdc: starting...
W20260618 23:34:14.515676 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:34:14 dist-test-slave-6224 krb5kdc[13991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:34:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:34:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:34:14.644490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33837
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:34:14.738312 14004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:14.738585 14004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:14.738612 14004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:14.740232 14004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:34:14.740299 14004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:14.740316 14004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:14.740331 14004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:34:14.740345 14004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:34:14.742101 14004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:14.742440 14004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:14.742741 14004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:14.748361 14012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:14.748361 14010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:14.748471 14009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:14.748567 14004 server_base.cc:1034] running on GCE node
I20260618 23:34:14.749050 14004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:14.749388 14004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:14.750561 14004 hybrid_clock.cc:648] HybridClock initialized: now 1781825654750542 us; error 37 us; skew 500 ppm
Jun 18 23:34:14 dist-test-slave-6224 krb5kdc[13991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:14.753609 14004 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:34:14.754172 14004 webserver.cc:460] Webserver started at http://127.0.98.62:37773/ using document root <none> and password file <none>
I20260618 23:34:14.754467 14004 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:14.754544 14004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:14.754712 14004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:34:14.756002 14004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "fb9affbad95f4f99822aa22be5368158"
format_stamp: "Formatted at 2026-06-18 23:34:14 on dist-test-slave-6224"
server_key: "8e18a0723b73595e4b99900b554fcb07"
server_key_iv: "464aeef42ebed7de1fb06015f25a925e"
server_key_version: "encryptionkey@0"
I20260618 23:34:14.756383 14004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb9affbad95f4f99822aa22be5368158"
format_stamp: "Formatted at 2026-06-18 23:34:14 on dist-test-slave-6224"
server_key: "8e18a0723b73595e4b99900b554fcb07"
server_key_iv: "464aeef42ebed7de1fb06015f25a925e"
server_key_version: "encryptionkey@0"
I20260618 23:34:14.758003 14004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:34:14.758986 14019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:14.759294 14004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:14.759410 14004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "fb9affbad95f4f99822aa22be5368158"
format_stamp: "Formatted at 2026-06-18 23:34:14 on dist-test-slave-6224"
server_key: "8e18a0723b73595e4b99900b554fcb07"
server_key_iv: "464aeef42ebed7de1fb06015f25a925e"
server_key_version: "encryptionkey@0"
I20260618 23:34:14.759492 14004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:14.772171 14004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:14.774544 14004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:14.774727 14004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:14.779173 14004 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33837
I20260618 23:34:14.779233 14071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33837 every 8 connection(s)
I20260618 23:34:14.779675 14004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:34:14.780157 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 14004
I20260618 23:34:14.780248 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:34:14.780469 392 external_mini_cluster.cc:1428] Setting key a4328a581159737461b3ba217f65e12d
I20260618 23:34:14.780952 14072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:34:14.784456 14072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158: Bootstrap starting.
I20260618 23:34:14.785171 14072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158: Neither blocks nor log segments found. Creating new log.
I20260618 23:34:14.785507 14072 log.cc:824] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:14.786465 14072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158: No bootstrap required, opened a new log
Jun 18 23:34:14 dist-test-slave-6224 krb5kdc[13991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:34:14.789218 14072 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9affbad95f4f99822aa22be5368158" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33837 } }
I20260618 23:34:14.789397 14072 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:34:14.789450 14072 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb9affbad95f4f99822aa22be5368158, State: Initialized, Role: FOLLOWER
I20260618 23:34:14.789584 14072 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9affbad95f4f99822aa22be5368158" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33837 } }
I20260618 23:34:14.789680 14072 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:34:14.789742 14072 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:34:14.789794 14072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:34:14.790527 14072 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9affbad95f4f99822aa22be5368158" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33837 } }
I20260618 23:34:14.790658 14072 leader_election.cc:302] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb9affbad95f4f99822aa22be5368158; no voters:
I20260618 23:34:14.790882 14072 leader_election.cc:288] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:34:14.790964 14077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:34:14.791218 14072 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:34:14.791241 14077 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158 [term 1 LEADER]: Becoming Leader. State: Replica: fb9affbad95f4f99822aa22be5368158, State: Running, Role: LEADER
I20260618 23:34:14.791353 14077 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9affbad95f4f99822aa22be5368158" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33837 } }
I20260618 23:34:14.791833 14078 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb9affbad95f4f99822aa22be5368158" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9affbad95f4f99822aa22be5368158" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33837 } } }
I20260618 23:34:14.791922 14078 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158 [sys.catalog]: This master's current role is: LEADER
I20260618 23:34:14.792181 14079 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb9affbad95f4f99822aa22be5368158. Latest consensus state: current_term: 1 leader_uuid: "fb9affbad95f4f99822aa22be5368158" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb9affbad95f4f99822aa22be5368158" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33837 } } }
I20260618 23:34:14.792316 14079 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158 [sys.catalog]: This master's current role is: LEADER
I20260618 23:34:14.792654 14086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:34:14.793049 14075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:14.781238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52594 (local address 127.0.98.62:33837)
0618 23:34:14.781448 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:14.781457 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:14.781483 (+ 26us) server_negotiation.cc:407] Connection header received
0618 23:34:14.782362 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:14.782371 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:14.782425 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:14.782600 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:14.783224 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:14.784262 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:14.785076 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:14.785548 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:14.785671 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:14.788754 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:14.788765 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:14.788765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:14.788779 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:14.790743 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:14.791271 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:14.791275 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:14.791276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:14.791396 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:14.791715 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:14.791719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:14.791719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:14.792139 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:14.792299 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:14.792629 (+ 330us) server_negotiation.cc:299] Negotiation successful
0618 23:34:14.792822 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":162,"thread_start_us":103,"threads_started":1}
I20260618 23:34:14.793574 14086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:34:14.794155 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:34:14.795387 14086 catalog_manager.cc:1347] Generated new cluster ID: 55e6579b99674ea5a24744cb4f410863
I20260618 23:34:14.795440 14086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:34:14.799602 14086 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:34:14.800148 14086 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:34:14.804088 14086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb9affbad95f4f99822aa22be5368158: Generated new TSK 0
I20260618 23:34:14.804288 14086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:34:14.848850 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 14004
2026-06-18T23:34:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1379 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:34:14 dist-test-slave-6224 krb5kdc[14099](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:34:14 dist-test-slave-6224 krb5kdc[14099](info): set up 2 sockets
Jun 18 23:34:14 dist-test-slave-6224 krb5kdc[14099](info): commencing operation
krb5kdc: starting...
W20260618 23:34:15.887816 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:34:15 dist-test-slave-6224 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:34:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:34:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:34:16.027392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36203
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:34:16.119494 14112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:16.119783 14112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:16.119820 14112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:16.121032 14112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:34:16.121098 14112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:16.121122 14112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:16.121143 14112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:34:16.121172 14112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:34:16.124158 14112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:16.124537 14112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:16.124800 14112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:16.130015 14120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:16.130015 14117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:16.130177 14118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:16.130209 14112 server_base.cc:1034] running on GCE node
I20260618 23:34:16.130568 14112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:16.130950 14112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:16.132119 14112 hybrid_clock.cc:648] HybridClock initialized: now 1781825656132103 us; error 38 us; skew 500 ppm
Jun 18 23:34:16 dist-test-slave-6224 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:16.135036 14112 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:34:16.135771 14112 webserver.cc:460] Webserver started at http://127.0.98.62:34315/ using document root <none> and password file <none>
I20260618 23:34:16.136071 14112 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:16.136147 14112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:16.136318 14112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:34:16.137598 14112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "aa04e35ddeee494b89559a1f529a9470"
format_stamp: "Formatted at 2026-06-18 23:34:16 on dist-test-slave-6224"
server_key: "74b633b26516b7b6d0930fd738edf1e3"
server_key_iv: "77ea7ea6e14d5c248d61b55b4b8981a3"
server_key_version: "encryptionkey@0"
I20260618 23:34:16.138042 14112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "aa04e35ddeee494b89559a1f529a9470"
format_stamp: "Formatted at 2026-06-18 23:34:16 on dist-test-slave-6224"
server_key: "74b633b26516b7b6d0930fd738edf1e3"
server_key_iv: "77ea7ea6e14d5c248d61b55b4b8981a3"
server_key_version: "encryptionkey@0"
I20260618 23:34:16.139838 14112 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260618 23:34:16.140866 14127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:16.141116 14112 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:16.141220 14112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "aa04e35ddeee494b89559a1f529a9470"
format_stamp: "Formatted at 2026-06-18 23:34:16 on dist-test-slave-6224"
server_key: "74b633b26516b7b6d0930fd738edf1e3"
server_key_iv: "77ea7ea6e14d5c248d61b55b4b8981a3"
server_key_version: "encryptionkey@0"
I20260618 23:34:16.141316 14112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:16.151630 14112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:16.154385 14112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:16.154579 14112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:16.159704 14112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36203
I20260618 23:34:16.159777 14179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36203 every 8 connection(s)
I20260618 23:34:16.160213 14112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:34:16.161463 14180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:34:16.162926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 14112
I20260618 23:34:16.163062 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:34:16.163277 392 external_mini_cluster.cc:1428] Setting key 5e9c19984f3c9d9cfab925fd12c7dbc9
I20260618 23:34:16.164784 14180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470: Bootstrap starting.
I20260618 23:34:16.165654 14180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470: Neither blocks nor log segments found. Creating new log.
I20260618 23:34:16.166049 14180 log.cc:824] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:16.166894 14180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470: No bootstrap required, opened a new log
I20260618 23:34:16.169696 14180 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa04e35ddeee494b89559a1f529a9470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } }
I20260618 23:34:16.169922 14180 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:34:16.169968 14180 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa04e35ddeee494b89559a1f529a9470, State: Initialized, Role: FOLLOWER
I20260618 23:34:16.170138 14180 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa04e35ddeee494b89559a1f529a9470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } }
I20260618 23:34:16.170243 14180 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:34:16.170284 14180 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:34:16.170358 14180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:34:16.171257 14180 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa04e35ddeee494b89559a1f529a9470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } }
I20260618 23:34:16.171356 14180 leader_election.cc:302] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa04e35ddeee494b89559a1f529a9470; no voters:
I20260618 23:34:16.171623 14180 leader_election.cc:288] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:34:16.171649 14185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:34:16.171782 14185 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 1 LEADER]: Becoming Leader. State: Replica: aa04e35ddeee494b89559a1f529a9470, State: Running, Role: LEADER
Jun 18 23:34:16 dist-test-slave-6224 krb5kdc[14099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:34:16.171985 14180 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:34:16.171996 14185 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa04e35ddeee494b89559a1f529a9470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } }
I20260618 23:34:16.172529 14186 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa04e35ddeee494b89559a1f529a9470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa04e35ddeee494b89559a1f529a9470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } } }
I20260618 23:34:16.172582 14187 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa04e35ddeee494b89559a1f529a9470. Latest consensus state: current_term: 1 leader_uuid: "aa04e35ddeee494b89559a1f529a9470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa04e35ddeee494b89559a1f529a9470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } } }
I20260618 23:34:16.172706 14186 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [sys.catalog]: This master's current role is: LEADER
I20260618 23:34:16.173023 14187 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [sys.catalog]: This master's current role is: LEADER
I20260618 23:34:16.174055 14194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:34:16.174690 14194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:34:16.176785 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:16.164054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40254 (local address 127.0.98.62:36203)
0618 23:34:16.164269 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:16.164281 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:16.164318 (+ 37us) server_negotiation.cc:407] Connection header received
0618 23:34:16.164908 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:16.164919 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:16.164988 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:16.165145 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:16.166079 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.167223 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.168452 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.168959 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.169098 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:16.172496 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:16.172501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:16.172502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:16.172535 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:16.174993 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:16.175435 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:16.175438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:16.175438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:16.175541 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:16.175784 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:16.175786 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:16.175786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:16.176145 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:16.176336 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:16.176421 (+ 85us) server_negotiation.cc:299] Negotiation successful
0618 23:34:16.176587 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":117,"threads_started":1}
I20260618 23:34:16.177443 14194 catalog_manager.cc:1347] Generated new cluster ID: 2de2c26b95d0421f8679603ec886707a
I20260618 23:34:16.177508 14194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:34:16.207165 14194 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:34:16.208217 14194 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:34:16.223651 14194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470: Generated new TSK 0
I20260618 23:34:16.223964 14194 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260618 23:34:16.252699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36203
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:34:16.340773 14206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:16.341003 14206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:16.341029 14206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:16.342175 14206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:16.342226 14206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:16.342253 14206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:34:16.343767 14206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:16.344027 14206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:16.344269 14206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:16.349761 14211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:16.349777 14214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:16.349799 14212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:16.349975 14206 server_base.cc:1034] running on GCE node
I20260618 23:34:16.350358 14206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:16.350668 14206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:16.351821 14206 hybrid_clock.cc:648] HybridClock initialized: now 1781825656351783 us; error 53 us; skew 500 ppm
Jun 18 23:34:16 dist-test-slave-6224 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:16.354743 14206 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:34:16.355278 14206 webserver.cc:460] Webserver started at http://127.0.98.1:37065/ using document root <none> and password file <none>
I20260618 23:34:16.355585 14206 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:16.355649 14206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:16.355813 14206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:34:16.357038 14206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "efa7e30dcd9841d993d31ba6b984fd0a"
format_stamp: "Formatted at 2026-06-18 23:34:16 on dist-test-slave-6224"
server_key: "59b3186c3534a30250b8c69852037fa1"
server_key_iv: "80c5e5339cff7b370a5b003e7cf66109"
server_key_version: "encryptionkey@0"
I20260618 23:34:16.357466 14206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "efa7e30dcd9841d993d31ba6b984fd0a"
format_stamp: "Formatted at 2026-06-18 23:34:16 on dist-test-slave-6224"
server_key: "59b3186c3534a30250b8c69852037fa1"
server_key_iv: "80c5e5339cff7b370a5b003e7cf66109"
server_key_version: "encryptionkey@0"
I20260618 23:34:16.358948 14206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:34:16.359849 14221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:16.360126 14206 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:16.360211 14206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "efa7e30dcd9841d993d31ba6b984fd0a"
format_stamp: "Formatted at 2026-06-18 23:34:16 on dist-test-slave-6224"
server_key: "59b3186c3534a30250b8c69852037fa1"
server_key_iv: "80c5e5339cff7b370a5b003e7cf66109"
server_key_version: "encryptionkey@0"
I20260618 23:34:16.360304 14206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:16.398079 14206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:16.400345 14206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:16.400471 14206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:16.400678 14206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:34:16.401041 14206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:34:16.401091 14206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:16.401117 14206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:34:16.401168 14206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:16.408069 14206 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44499
I20260618 23:34:16.408147 14334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44499 every 8 connection(s)
I20260618 23:34:16.408528 14206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
Jun 18 23:34:16 dist-test-slave-6224 krb5kdc[14099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:34:16.418489 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 14206
I20260618 23:34:16.418628 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:34:16.418918 392 external_mini_cluster.cc:1428] Setting key 739932461f1e89287a92ecb27829558b
I20260618 23:34:16.419062 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:16.409516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49515 (local address 127.0.98.62:36203)
0618 23:34:16.409592 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:16.409594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:16.409768 (+ 174us) server_negotiation.cc:407] Connection header received
0618 23:34:16.410300 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:16.410301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:16.410316 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:16.410407 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:16.411455 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.412112 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.413159 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.413381 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.413463 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:16.415824 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:16.415826 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:16.415826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:16.415833 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:16.417500 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:16.417916 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:16.417916 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:16.417917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:16.417973 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:16.418220 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:16.418257 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:16.418258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:16.418433 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:16.418523 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:16.418851 (+ 328us) server_negotiation.cc:299] Negotiation successful
0618 23:34:16.418957 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260618 23:34:16.419253 14337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:16.409447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.1:49515)
0618 23:34:16.409614 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:16.409660 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:16.410188 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:16.410494 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:16.410498 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:16.411114 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:16.411315 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:16.411320 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.412228 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.412229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:16.413057 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:16.413058 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.413477 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.413478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:16.413579 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:16.414131 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:16.414139 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:16.415730 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:16.417580 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:16.417586 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:16.417588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:16.417841 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:16.418039 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:16.418039 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:16.418040 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:16.418139 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:16.418514 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:16.418518 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:16.418703 (+ 185us) client_negotiation.cc:769] Sending connection context
0618 23:34:16.418814 (+ 111us) client_negotiation.cc:240] Negotiation successful
0618 23:34:16.419001 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260618 23:34:16.419984 14335 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36203
I20260618 23:34:16.420127 14335 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:16.420363 14335 heartbeater.cc:511] Master 127.0.98.62:36203 requested a full tablet report, sending...
I20260618 23:34:16.421056 14144 ts_manager.cc:194] Registered new tserver with Master: efa7e30dcd9841d993d31ba6b984fd0a (127.0.98.1:44499)
I20260618 23:34:16.422065 14144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49515
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260618 23:34:16.446184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36203
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:34:16.538928 14340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:16.539223 14340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:16.539256 14340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:16.540743 14340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:16.540812 14340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:16.540848 14340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:34:16.542520 14340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:16.542846 14340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:16.543144 14340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:16.549660 14346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:16.549664 14348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:16.549621 14345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:16.549855 14340 server_base.cc:1034] running on GCE node
I20260618 23:34:16.550105 14340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:16.550441 14340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:16.551607 14340 hybrid_clock.cc:648] HybridClock initialized: now 1781825656551590 us; error 44 us; skew 500 ppm
Jun 18 23:34:16 dist-test-slave-6224 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:16.554556 14340 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:34:16.555073 14340 webserver.cc:460] Webserver started at http://127.0.98.2:36367/ using document root <none> and password file <none>
I20260618 23:34:16.555337 14340 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:16.555406 14340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:16.555632 14340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:34:16.556797 14340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "be6c382c0e6f45a5b00483b7291bbd19"
format_stamp: "Formatted at 2026-06-18 23:34:16 on dist-test-slave-6224"
server_key: "e10b1e9723185edcf446c300ea3989bf"
server_key_iv: "6d1ca22207a2f1b82b6e5b47465b7eaa"
server_key_version: "encryptionkey@0"
I20260618 23:34:16.557170 14340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "be6c382c0e6f45a5b00483b7291bbd19"
format_stamp: "Formatted at 2026-06-18 23:34:16 on dist-test-slave-6224"
server_key: "e10b1e9723185edcf446c300ea3989bf"
server_key_iv: "6d1ca22207a2f1b82b6e5b47465b7eaa"
server_key_version: "encryptionkey@0"
I20260618 23:34:16.558640 14340 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:34:16.559508 14355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:16.559829 14340 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:16.559943 14340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "be6c382c0e6f45a5b00483b7291bbd19"
format_stamp: "Formatted at 2026-06-18 23:34:16 on dist-test-slave-6224"
server_key: "e10b1e9723185edcf446c300ea3989bf"
server_key_iv: "6d1ca22207a2f1b82b6e5b47465b7eaa"
server_key_version: "encryptionkey@0"
I20260618 23:34:16.560024 14340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:16.567482 14340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:16.569862 14340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:16.570055 14340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:16.570276 14340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:34:16.570587 14340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:34:16.570641 14340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:16.570683 14340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:34:16.570709 14340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:16.577515 14340 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39423
I20260618 23:34:16.577740 14468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39423 every 8 connection(s)
I20260618 23:34:16.577955 14340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:34:16.582106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 14340
I20260618 23:34:16.582228 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:34:16.582518 392 external_mini_cluster.cc:1428] Setting key cb2134bd093274f6de6ce92ac013a395
Jun 18 23:34:16 dist-test-slave-6224 krb5kdc[14099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:34:16.588421 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:16.578812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37191 (local address 127.0.98.62:36203)
0618 23:34:16.578877 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:16.578879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:16.579067 (+ 188us) server_negotiation.cc:407] Connection header received
0618 23:34:16.579669 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:16.579670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:16.579683 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:16.579770 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:16.580750 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.581326 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.582442 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.582763 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.582848 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:16.585112 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:16.585114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:16.585114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:16.585121 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:16.586622 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:16.587152 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:16.587153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:16.587154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:16.587222 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:16.587522 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:16.587523 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:16.587523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:16.587751 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:16.587835 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:16.588249 (+ 414us) server_negotiation.cc:299] Negotiation successful
0618 23:34:16.588341 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260618 23:34:16.588677 14471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:16.578753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.2:37191)
0618 23:34:16.578928 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:16.578971 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:16.579456 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:16.579843 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:16.579846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:16.580444 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:16.580642 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:16.580646 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.581420 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.581421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:16.582341 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:16.582344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.582854 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.582855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:16.582991 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:16.583597 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:16.583606 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:16.585004 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:16.586710 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:16.586716 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:16.586718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:16.587052 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:16.587302 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:16.587303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:16.587304 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:16.587447 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:16.587844 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:16.587852 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:16.588049 (+ 197us) client_negotiation.cc:769] Sending connection context
0618 23:34:16.588190 (+ 141us) client_negotiation.cc:240] Negotiation successful
0618 23:34:16.588375 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260618 23:34:16.589362 14469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36203
I20260618 23:34:16.589534 14469 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:16.589792 14469 heartbeater.cc:511] Master 127.0.98.62:36203 requested a full tablet report, sending...
I20260618 23:34:16.590346 14144 ts_manager.cc:194] Registered new tserver with Master: be6c382c0e6f45a5b00483b7291bbd19 (127.0.98.2:39423)
I20260618 23:34:16.590937 14144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37191
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260618 23:34:16.610543 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36203
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:34:16.697956 14474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:16.698172 14474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:16.698202 14474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:16.699321 14474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:16.699371 14474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:16.699404 14474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:34:16.700932 14474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:16.701188 14474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:16.701438 14474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:16.706777 14479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:16.706812 14480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:16.706799 14482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:16.707222 14474 server_base.cc:1034] running on GCE node
I20260618 23:34:16.707425 14474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:16.707731 14474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:16.708881 14474 hybrid_clock.cc:648] HybridClock initialized: now 1781825656708870 us; error 28 us; skew 500 ppm
Jun 18 23:34:16 dist-test-slave-6224 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:16.711688 14474 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:34:16.712234 14474 webserver.cc:460] Webserver started at http://127.0.98.3:45423/ using document root <none> and password file <none>
I20260618 23:34:16.712513 14474 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:16.712563 14474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:16.712723 14474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:34:16.713973 14474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "0ebf2f83a0264e7cb5257abb7942a1ec"
format_stamp: "Formatted at 2026-06-18 23:34:16 on dist-test-slave-6224"
server_key: "24fdfc0998be3058d2eaca70e24d66f0"
server_key_iv: "261bd1ef601f4d73ef97c4d43442d216"
server_key_version: "encryptionkey@0"
I20260618 23:34:16.714380 14474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0ebf2f83a0264e7cb5257abb7942a1ec"
format_stamp: "Formatted at 2026-06-18 23:34:16 on dist-test-slave-6224"
server_key: "24fdfc0998be3058d2eaca70e24d66f0"
server_key_iv: "261bd1ef601f4d73ef97c4d43442d216"
server_key_version: "encryptionkey@0"
I20260618 23:34:16.716045 14474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.003s
I20260618 23:34:16.717178 14489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:16.717545 14474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:16.717689 14474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "0ebf2f83a0264e7cb5257abb7942a1ec"
format_stamp: "Formatted at 2026-06-18 23:34:16 on dist-test-slave-6224"
server_key: "24fdfc0998be3058d2eaca70e24d66f0"
server_key_iv: "261bd1ef601f4d73ef97c4d43442d216"
server_key_version: "encryptionkey@0"
I20260618 23:34:16.717814 14474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:16.727425 14474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:16.729771 14474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:16.729897 14474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:16.730129 14474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:34:16.730727 14474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:34:16.730772 14474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:16.730799 14474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:34:16.730813 14474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:16.738621 14474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45521
I20260618 23:34:16.738719 14602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45521 every 8 connection(s)
I20260618 23:34:16.739097 14474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
Jun 18 23:34:16 dist-test-slave-6224 krb5kdc[14099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:34:16.746649 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 14474
I20260618 23:34:16.746768 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:34:16.746964 392 external_mini_cluster.cc:1428] Setting key 0ed7d623b2941a72f8c0e05ac8674cda
I20260618 23:34:16.749884 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:16.740068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41437 (local address 127.0.98.62:36203)
0618 23:34:16.740152 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:16.740155 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:16.740408 (+ 253us) server_negotiation.cc:407] Connection header received
0618 23:34:16.740964 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:16.740965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:16.740976 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:16.741034 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:16.742147 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.742806 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.744234 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.744525 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.744608 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:16.747066 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:16.747068 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:16.747069 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:16.747079 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:16.748509 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:16.748930 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:16.748931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:16.748932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:16.748986 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:16.749227 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:16.749228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:16.749228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:16.749366 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:16.749440 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:16.749705 (+ 265us) server_negotiation.cc:299] Negotiation successful
0618 23:34:16.749786 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 23:34:16.750092 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:16.740026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.3:41437)
0618 23:34:16.740253 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:16.740298 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:16.740865 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:16.741108 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:16.741112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:16.741767 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:16.742043 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:16.742049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.742924 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.742924 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:16.743984 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:16.743986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.744641 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.744642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:16.744814 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:16.745478 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:16.745486 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:16.746955 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:16.748608 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:16.748615 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:16.748617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:16.748832 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:16.749049 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:16.749050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:16.749051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:16.749163 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:16.749433 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:16.749439 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:16.749576 (+ 137us) client_negotiation.cc:769] Sending connection context
0618 23:34:16.749680 (+ 104us) client_negotiation.cc:240] Negotiation successful
0618 23:34:16.749833 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":120,"threads_started":1}
I20260618 23:34:16.750679 14603 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36203
I20260618 23:34:16.750765 14603 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:16.750983 14603 heartbeater.cc:511] Master 127.0.98.62:36203 requested a full tablet report, sending...
I20260618 23:34:16.751430 14144 ts_manager.cc:194] Registered new tserver with Master: 0ebf2f83a0264e7cb5257abb7942a1ec (127.0.98.3:45521)
I20260618 23:34:16.751914 14144 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41437
I20260618 23:34:16.758939 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:34:16.768261 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:16.761233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40258 (local address 127.0.98.62:36203)
0618 23:34:16.761317 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:16.761319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:16.761331 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:34:16.761388 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:16.761389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:16.761401 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:16.761459 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:16.762111 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.762807 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.763571 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.763876 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.763956 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:16.765320 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:16.765322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:16.765323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:16.765334 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:16.767062 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:16.767439 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:16.767439 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:16.767440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:16.767492 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:16.767714 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:16.767714 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:16.767715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:16.767881 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:16.767966 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:16.768124 (+ 158us) server_negotiation.cc:299] Negotiation successful
0618 23:34:16.768194 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 23:34:16.770048 14144 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40258:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:34:16.772089 14144 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:34:16.779445 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:16.775127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44499 (local address 127.0.0.1:54278)
0618 23:34:16.775585 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:16.775598 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:16.775648 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:16.775854 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:16.775857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:16.775859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:34:16.775956 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:16.775960 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.777053 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.777054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:16.778341 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:16.778344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.779110 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.779111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:16.779196 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:16.779210 (+ 14us) client_negotiation.cc:769] Sending connection context
0618 23:34:16.779302 (+ 92us) client_negotiation.cc:240] Negotiation successful
0618 23:34:16.779375 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":156,"threads_started":1}
I20260618 23:34:16.779456 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:16.775384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54278 (local address 127.0.98.1:44499)
0618 23:34:16.775682 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:16.775688 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:16.775698 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:34:16.775736 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:16.775738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:16.775752 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:16.775838 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:34:16.776047 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.776959 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.778438 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.779027 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.779109 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:16.779167 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:16.779322 (+ 155us) server_negotiation.cc:299] Negotiation successful
0618 23:34:16.779362 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":135,"threads_started":1}
I20260618 23:34:16.779877 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:16.775088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45521 (local address 127.0.0.1:51254)
0618 23:34:16.775201 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:16.775240 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:16.775332 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:16.775902 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:16.775904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:16.775908 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:34:16.775996 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:16.775999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.777476 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.777478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:16.778593 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:16.778595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.779580 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.779581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:16.779673 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:16.779677 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:34:16.779713 (+ 36us) client_negotiation.cc:240] Negotiation successful
0618 23:34:16.779776 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":66,"thread_start_us":66,"threads_started":1}
I20260618 23:34:16.779979 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:16.775715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52304 (local address 127.0.98.2:39423)
0618 23:34:16.776178 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:16.776183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:16.776193 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:34:16.776236 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:16.776238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:16.776252 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:16.776343 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:34:16.776488 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.777518 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.778736 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.779445 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.779521 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:16.779615 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:16.779829 (+ 214us) server_negotiation.cc:299] Negotiation successful
0618 23:34:16.779879 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":98,"threads_started":1}
I20260618 23:34:16.780156 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:16.775401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39423 (local address 127.0.0.1:52304)
0618 23:34:16.775940 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:16.775951 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:16.776001 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:16.776359 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:16.776360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:16.776362 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:34:16.776412 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:16.776413 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.777629 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.777630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:16.778617 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:16.778619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.779646 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.779647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:16.779704 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:16.779708 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:34:16.779755 (+ 47us) client_negotiation.cc:240] Negotiation successful
0618 23:34:16.779790 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":101,"threads_started":1}
I20260618 23:34:16.780313 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:16.775368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51254 (local address 127.0.98.3:45521)
0618 23:34:16.775664 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:16.775670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:16.775684 (+ 14us) server_negotiation.cc:407] Connection header received
0618 23:34:16.775742 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:16.775745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:16.775766 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:16.776076 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:34:16.776190 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.777350 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.778690 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.779445 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.780033 (+ 588us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:16.780103 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:16.780129 (+ 26us) server_negotiation.cc:299] Negotiation successful
0618 23:34:16.780179 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":133,"threads_started":1}
I20260618 23:34:16.780428 14269 tablet_service.cc:1467] Processing CreateTablet for tablet 7178ed09ddda4c8fb36da19a703e2773 (DEFAULT_TABLE table=test-table [id=d4781e5320074ec7bd046facae9f7b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:34:16.780807 14269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7178ed09ddda4c8fb36da19a703e2773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:34:16.781106 14537 tablet_service.cc:1467] Processing CreateTablet for tablet 7178ed09ddda4c8fb36da19a703e2773 (DEFAULT_TABLE table=test-table [id=d4781e5320074ec7bd046facae9f7b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:34:16.781106 14403 tablet_service.cc:1467] Processing CreateTablet for tablet 7178ed09ddda4c8fb36da19a703e2773 (DEFAULT_TABLE table=test-table [id=d4781e5320074ec7bd046facae9f7b6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:34:16.781486 14403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7178ed09ddda4c8fb36da19a703e2773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:34:16.781486 14537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7178ed09ddda4c8fb36da19a703e2773. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:34:16.783834 14620 tablet_bootstrap.cc:492] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a: Bootstrap starting.
I20260618 23:34:16.784403 14620 tablet_bootstrap.cc:654] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a: Neither blocks nor log segments found. Creating new log.
I20260618 23:34:16.784894 14620 log.cc:824] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:16.785398 14622 tablet_bootstrap.cc:492] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec: Bootstrap starting.
I20260618 23:34:16.785637 14623 tablet_bootstrap.cc:492] T 7178ed09ddda4c8fb36da19a703e2773 P be6c382c0e6f45a5b00483b7291bbd19: Bootstrap starting.
I20260618 23:34:16.785688 14620 tablet_bootstrap.cc:492] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a: No bootstrap required, opened a new log
I20260618 23:34:16.785765 14620 ts_tablet_manager.cc:1397] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:34:16.786247 14622 tablet_bootstrap.cc:654] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec: Neither blocks nor log segments found. Creating new log.
I20260618 23:34:16.786332 14623 tablet_bootstrap.cc:654] T 7178ed09ddda4c8fb36da19a703e2773 P be6c382c0e6f45a5b00483b7291bbd19: Neither blocks nor log segments found. Creating new log.
I20260618 23:34:16.786954 14623 log.cc:824] T 7178ed09ddda4c8fb36da19a703e2773 P be6c382c0e6f45a5b00483b7291bbd19: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:16.786955 14622 log.cc:824] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:16.787417 14620 raft_consensus.cc:348] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } }
I20260618 23:34:16.787667 14620 raft_consensus.cc:374] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:34:16.787705 14620 raft_consensus.cc:729] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa7e30dcd9841d993d31ba6b984fd0a, State: Initialized, Role: FOLLOWER
I20260618 23:34:16.787787 14623 tablet_bootstrap.cc:492] T 7178ed09ddda4c8fb36da19a703e2773 P be6c382c0e6f45a5b00483b7291bbd19: No bootstrap required, opened a new log
I20260618 23:34:16.787870 14623 ts_tablet_manager.cc:1397] T 7178ed09ddda4c8fb36da19a703e2773 P be6c382c0e6f45a5b00483b7291bbd19: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:34:16.787832 14620 consensus_queue.cc:260] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } }
I20260618 23:34:16.787918 14622 tablet_bootstrap.cc:492] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec: No bootstrap required, opened a new log
I20260618 23:34:16.788000 14622 ts_tablet_manager.cc:1397] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 23:34:16.788146 14620 ts_tablet_manager.cc:1428] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:34:16.788235 14335 heartbeater.cc:503] Master 127.0.98.62:36203 was elected leader, sending a full tablet report...
I20260618 23:34:16.789808 14622 raft_consensus.cc:348] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } }
I20260618 23:34:16.789933 14622 raft_consensus.cc:374] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:34:16.789952 14622 raft_consensus.cc:729] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ebf2f83a0264e7cb5257abb7942a1ec, State: Initialized, Role: FOLLOWER
I20260618 23:34:16.790060 14622 consensus_queue.cc:260] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } }
I20260618 23:34:16.790064 14623 raft_consensus.cc:348] T 7178ed09ddda4c8fb36da19a703e2773 P be6c382c0e6f45a5b00483b7291bbd19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } }
I20260618 23:34:16.790215 14623 raft_consensus.cc:374] T 7178ed09ddda4c8fb36da19a703e2773 P be6c382c0e6f45a5b00483b7291bbd19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:34:16.790237 14623 raft_consensus.cc:729] T 7178ed09ddda4c8fb36da19a703e2773 P be6c382c0e6f45a5b00483b7291bbd19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be6c382c0e6f45a5b00483b7291bbd19, State: Initialized, Role: FOLLOWER
I20260618 23:34:16.790340 14622 ts_tablet_manager.cc:1428] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260618 23:34:16.790361 14623 consensus_queue.cc:260] T 7178ed09ddda4c8fb36da19a703e2773 P be6c382c0e6f45a5b00483b7291bbd19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } }
I20260618 23:34:16.790416 14603 heartbeater.cc:503] Master 127.0.98.62:36203 was elected leader, sending a full tablet report...
I20260618 23:34:16.790681 14623 ts_tablet_manager.cc:1428] T 7178ed09ddda4c8fb36da19a703e2773 P be6c382c0e6f45a5b00483b7291bbd19: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260618 23:34:16.790817 14469 heartbeater.cc:503] Master 127.0.98.62:36203 was elected leader, sending a full tablet report...
W20260618 23:34:16.828631 14470 tablet.cc:2369] T 7178ed09ddda4c8fb36da19a703e2773 P be6c382c0e6f45a5b00483b7291bbd19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:34:16.832686 14627 raft_consensus.cc:482] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:34:16.832844 14627 raft_consensus.cc:504] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } }
I20260618 23:34:16.833308 14627 leader_election.cc:288] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efa7e30dcd9841d993d31ba6b984fd0a (127.0.98.1:44499), be6c382c0e6f45a5b00483b7291bbd19 (127.0.98.2:39423)
I20260618 23:34:16.837556 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:16.833666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57075 (local address 127.0.98.1:44499)
0618 23:34:16.833721 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:16.833723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:16.833734 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:34:16.833777 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:16.833778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:16.833793 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:16.833858 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:34:16.834150 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.834961 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.836563 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.837164 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.837251 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:16.837262 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:16.837434 (+ 172us) server_negotiation.cc:299] Negotiation successful
0618 23:34:16.837477 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260618 23:34:16.837556 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:16.833866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36285 (local address 127.0.98.2:39423)
0618 23:34:16.833922 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:16.833924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:16.833979 (+ 55us) server_negotiation.cc:407] Connection header received
0618 23:34:16.834071 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:16.834072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:16.834102 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:16.834170 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:34:16.834314 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.835007 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.836565 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.837164 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.837251 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:16.837262 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:16.837438 (+ 176us) server_negotiation.cc:299] Negotiation successful
0618 23:34:16.837477 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260618 23:34:16.837585 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:16.833536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44499 (local address 127.0.98.3:57075)
0618 23:34:16.833626 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:16.833640 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:16.833690 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:16.833931 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:16.833933 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:16.833936 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:34:16.834039 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:16.834042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.835069 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.835070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:16.836451 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:16.836455 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.837264 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.837265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:16.837350 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:16.837353 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:34:16.837413 (+ 60us) client_negotiation.cc:240] Negotiation successful
0618 23:34:16.837474 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":11}
I20260618 23:34:16.837595 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:16.833695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39423 (local address 127.0.98.3:36285)
0618 23:34:16.833907 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:16.833917 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:16.833989 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:16.834185 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:16.834186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:16.834188 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:34:16.834240 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:16.834241 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.835093 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.835093 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:16.836451 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:16.836455 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.837274 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.837275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:16.837345 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:16.837350 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:34:16.837413 (+ 63us) client_negotiation.cc:240] Negotiation successful
0618 23:34:16.837474 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":128,"thread_start_us":141,"threads_started":1}
I20260618 23:34:16.838049 14289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7178ed09ddda4c8fb36da19a703e2773" candidate_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" is_pre_election: true
I20260618 23:34:16.838249 14289 raft_consensus.cc:2454] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebf2f83a0264e7cb5257abb7942a1ec in term 0.
I20260618 23:34:16.838364 14423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7178ed09ddda4c8fb36da19a703e2773" candidate_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be6c382c0e6f45a5b00483b7291bbd19" is_pre_election: true
I20260618 23:34:16.838562 14423 raft_consensus.cc:2454] T 7178ed09ddda4c8fb36da19a703e2773 P be6c382c0e6f45a5b00483b7291bbd19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ebf2f83a0264e7cb5257abb7942a1ec in term 0.
I20260618 23:34:16.838562 14492 leader_election.cc:302] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebf2f83a0264e7cb5257abb7942a1ec, efa7e30dcd9841d993d31ba6b984fd0a; no voters:
I20260618 23:34:16.838737 14627 raft_consensus.cc:2781] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:34:16.838816 14627 raft_consensus.cc:482] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:34:16.838845 14627 raft_consensus.cc:3037] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:34:16.839711 14627 raft_consensus.cc:504] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } }
I20260618 23:34:16.839877 14627 leader_election.cc:288] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec [CANDIDATE]: Term 1 election: Requested vote from peers efa7e30dcd9841d993d31ba6b984fd0a (127.0.98.1:44499), be6c382c0e6f45a5b00483b7291bbd19 (127.0.98.2:39423)
I20260618 23:34:16.840099 14289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7178ed09ddda4c8fb36da19a703e2773" candidate_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efa7e30dcd9841d993d31ba6b984fd0a"
I20260618 23:34:16.840162 14423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7178ed09ddda4c8fb36da19a703e2773" candidate_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be6c382c0e6f45a5b00483b7291bbd19"
I20260618 23:34:16.840206 14289 raft_consensus.cc:3037] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:34:16.840299 14423 raft_consensus.cc:3037] T 7178ed09ddda4c8fb36da19a703e2773 P be6c382c0e6f45a5b00483b7291bbd19 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:34:16.841068 14289 raft_consensus.cc:2454] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebf2f83a0264e7cb5257abb7942a1ec in term 1.
I20260618 23:34:16.841233 14423 raft_consensus.cc:2454] T 7178ed09ddda4c8fb36da19a703e2773 P be6c382c0e6f45a5b00483b7291bbd19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ebf2f83a0264e7cb5257abb7942a1ec in term 1.
I20260618 23:34:16.841336 14492 leader_election.cc:302] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebf2f83a0264e7cb5257abb7942a1ec, efa7e30dcd9841d993d31ba6b984fd0a; no voters:
I20260618 23:34:16.841449 14627 raft_consensus.cc:2781] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:34:16.841631 14627 raft_consensus.cc:686] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec [term 1 LEADER]: Becoming Leader. State: Replica: 0ebf2f83a0264e7cb5257abb7942a1ec, State: Running, Role: LEADER
I20260618 23:34:16.841732 14627 consensus_queue.cc:237] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } }
I20260618 23:34:16.842751 14143 catalog_manager.cc:5495] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ebf2f83a0264e7cb5257abb7942a1ec (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } health_report { overall_health: UNKNOWN } } }
I20260618 23:34:16.854161 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:16.851613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51260 (local address 127.0.98.3:45521)
0618 23:34:16.851694 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:16.851696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:16.851704 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:34:16.851741 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:16.851742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:16.851765 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:16.851852 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:34:16.852004 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.852518 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.853295 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:16.853500 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:16.853586 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:16.853758 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:34:16.853884 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:34:16.853988 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:16.854005 (+ 17us) server_negotiation.cc:299] Negotiation successful
0618 23:34:16.854030 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 23:34:16.856361 14423 raft_consensus.cc:1261] T 7178ed09ddda4c8fb36da19a703e2773 P be6c382c0e6f45a5b00483b7291bbd19 [term 1 FOLLOWER]: Refusing update from remote peer 0ebf2f83a0264e7cb5257abb7942a1ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:34:16.856453 14289 raft_consensus.cc:1261] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Refusing update from remote peer 0ebf2f83a0264e7cb5257abb7942a1ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:34:16.856762 14627 consensus_queue.cc:1035] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:34:16.856916 14630 consensus_queue.cc:1035] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:34:16.859573 14633 mvcc.cc:205] Tried to move back new op lower bound from 7298357890480820224 to 7298357890424725504. Current Snapshot: MvccSnapshot[applied={T|T < 7298357890480820224}]
I20260618 23:34:16.859810 14635 mvcc.cc:205] Tried to move back new op lower bound from 7298357890480820224 to 7298357890424725504. Current Snapshot: MvccSnapshot[applied={T|T < 7298357890480820224}]
I20260618 23:34:16.862190 14144 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:34:16.862303 14144 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40258:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:34:16.863219 14144 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470: Sending DeleteTablet for 3 replicas of tablet 7178ed09ddda4c8fb36da19a703e2773
I20260618 23:34:16.863641 14269 tablet_service.cc:1514] Processing DeleteTablet for tablet 7178ed09ddda4c8fb36da19a703e2773 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:34:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54278
I20260618 23:34:16.863628 14537 tablet_service.cc:1514] Processing DeleteTablet for tablet 7178ed09ddda4c8fb36da19a703e2773 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:34:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51254
I20260618 23:34:16.863694 14403 tablet_service.cc:1514] Processing DeleteTablet for tablet 7178ed09ddda4c8fb36da19a703e2773 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:34:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52304
I20260618 23:34:16.863735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 14206
I20260618 23:34:16.864565 14644 tablet_replica.cc:330] T 7178ed09ddda4c8fb36da19a703e2773 P be6c382c0e6f45a5b00483b7291bbd19: stopping tablet replica
I20260618 23:34:16.864677 14644 raft_consensus.cc:2229] T 7178ed09ddda4c8fb36da19a703e2773 P be6c382c0e6f45a5b00483b7291bbd19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:34:16.864782 14644 raft_consensus.cc:2258] T 7178ed09ddda4c8fb36da19a703e2773 P be6c382c0e6f45a5b00483b7291bbd19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:34:16.865571 14643 tablet_replica.cc:330] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec: stopping tablet replica
I20260618 23:34:16.865705 14643 raft_consensus.cc:2229] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:34:16.865856 14643 raft_consensus.cc:2258] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:34:16.866348 14643 ts_tablet_manager.cc:1905] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:34:16.867023 14644 ts_tablet_manager.cc:1905] T 7178ed09ddda4c8fb36da19a703e2773 P be6c382c0e6f45a5b00483b7291bbd19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:34:16.868624 14643 ts_tablet_manager.cc:1918] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:34:16.868716 14643 log.cc:1196] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/7178ed09ddda4c8fb36da19a703e2773
I20260618 23:34:16.869033 14643 ts_tablet_manager.cc:1939] T 7178ed09ddda4c8fb36da19a703e2773 P 0ebf2f83a0264e7cb5257abb7942a1ec: Deleting consensus metadata
I20260618 23:34:16.869607 14131 catalog_manager.cc:4841] TS 0ebf2f83a0264e7cb5257abb7942a1ec (127.0.98.3:45521): tablet 7178ed09ddda4c8fb36da19a703e2773 (table test-table [id=d4781e5320074ec7bd046facae9f7b6f]) successfully deleted
I20260618 23:34:16.869817 14644 ts_tablet_manager.cc:1918] T 7178ed09ddda4c8fb36da19a703e2773 P be6c382c0e6f45a5b00483b7291bbd19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:34:16.869979 14644 log.cc:1196] T 7178ed09ddda4c8fb36da19a703e2773 P be6c382c0e6f45a5b00483b7291bbd19: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/7178ed09ddda4c8fb36da19a703e2773
I20260618 23:34:16.870357 14644 ts_tablet_manager.cc:1939] T 7178ed09ddda4c8fb36da19a703e2773 P be6c382c0e6f45a5b00483b7291bbd19: Deleting consensus metadata
W20260618 23:34:16.870437 14130 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44499 (error 108)
I20260618 23:34:16.870786 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 14340
I20260618 23:34:16.871482 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:16.870938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44499 (local address 127.0.0.1:54292)
0618 23:34:16.871396 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:16.871436 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433}
W20260618 23:34:16.876403 14130 catalog_manager.cc:4584] TS efa7e30dcd9841d993d31ba6b984fd0a (127.0.98.1:44499): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7178ed09ddda4c8fb36da19a703e2773: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44499: connect: Connection refused (error 111)
I20260618 23:34:16.878600 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:16.878470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39423 (local address 127.0.0.1:52314)
0618 23:34:16.878538 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:16.878569 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260618 23:34:16.878796 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 14474
I20260618 23:34:16.884970 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 14112
I20260618 23:34:16.891065 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36203
--webserver_interface=127.0.98.62
--webserver_port=34315
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36203
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:34:16.976599 14646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:16.976833 14646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:16.976859 14646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:16.978070 14646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:34:16.978119 14646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:16.978134 14646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:16.978147 14646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:34:16.978159 14646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:34:16.979687 14646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:16.979961 14646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:16.980219 14646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:16.989833 14652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:16.989849 14655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:16.989853 14653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:16.990281 14646 server_base.cc:1034] running on GCE node
I20260618 23:34:16.990605 14646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:16.990976 14646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:16.992169 14646 hybrid_clock.cc:648] HybridClock initialized: now 1781825656992142 us; error 41 us; skew 500 ppm
Jun 18 23:34:16 dist-test-slave-6224 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:16.994879 14646 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:34:16.995380 14646 webserver.cc:460] Webserver started at http://127.0.98.62:34315/ using document root <none> and password file <none>
I20260618 23:34:16.995678 14646 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:16.995741 14646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:16.997262 14646 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:34:16.998111 14662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:16.998371 14646 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:34:16.998502 14646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "aa04e35ddeee494b89559a1f529a9470"
format_stamp: "Formatted at 2026-06-18 23:34:16 on dist-test-slave-6224"
server_key: "74b633b26516b7b6d0930fd738edf1e3"
server_key_iv: "77ea7ea6e14d5c248d61b55b4b8981a3"
server_key_version: "encryptionkey@0"
I20260618 23:34:16.998966 14646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:17.016211 14646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:17.019166 14646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:17.019369 14646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:17.024863 14646 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36203
I20260618 23:34:17.025032 14714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36203 every 8 connection(s)
I20260618 23:34:17.025447 14646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:34:17.027325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 14646
I20260618 23:34:17.027417 14715 sys_catalog.cc:263] Verifying existing consensus state
I20260618 23:34:17.027709 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44499
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36203
--webserver_port=37065
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:34:17.028332 14715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470: Bootstrap starting.
I20260618 23:34:17.037541 14715 log.cc:824] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:17.039788 14715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:34:17.040104 14715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470: Bootstrap complete.
I20260618 23:34:17.042724 14715 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa04e35ddeee494b89559a1f529a9470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } }
I20260618 23:34:17.043004 14715 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa04e35ddeee494b89559a1f529a9470, State: Initialized, Role: FOLLOWER
I20260618 23:34:17.043220 14715 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa04e35ddeee494b89559a1f529a9470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } }
I20260618 23:34:17.043322 14715 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:34:17.043385 14715 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:34:17.043438 14715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 1 FOLLOWER]: Advancing to term 2
I20260618 23:34:17.044571 14715 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa04e35ddeee494b89559a1f529a9470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } }
I20260618 23:34:17.044698 14715 leader_election.cc:302] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa04e35ddeee494b89559a1f529a9470; no voters:
I20260618 23:34:17.044920 14715 leader_election.cc:288] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260618 23:34:17.044962 14719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 2 FOLLOWER]: Leader election won for term 2
I20260618 23:34:17.045096 14719 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 2 LEADER]: Becoming Leader. State: Replica: aa04e35ddeee494b89559a1f529a9470, State: Running, Role: LEADER
I20260618 23:34:17.045218 14715 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:34:17.045234 14719 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa04e35ddeee494b89559a1f529a9470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } }
I20260618 23:34:17.045616 14721 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "aa04e35ddeee494b89559a1f529a9470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa04e35ddeee494b89559a1f529a9470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } } }
I20260618 23:34:17.045770 14721 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [sys.catalog]: This master's current role is: LEADER
I20260618 23:34:17.046084 14720 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa04e35ddeee494b89559a1f529a9470. Latest consensus state: current_term: 2 leader_uuid: "aa04e35ddeee494b89559a1f529a9470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa04e35ddeee494b89559a1f529a9470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } } }
I20260618 23:34:17.046176 14720 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [sys.catalog]: This master's current role is: LEADER
I20260618 23:34:17.046416 14728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:34:17.047397 14728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:34:17.047509 14728 catalog_manager.cc:1259] Loaded cluster ID: 2de2c26b95d0421f8679603ec886707a
I20260618 23:34:17.047591 14728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:34:17.048269 14728 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260618 23:34:17.053550 14728 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260618 23:34:17.056600 14728 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:34:17.056867 14728 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470: Loaded TSK: 0
I20260618 23:34:17.057132 14728 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 23:34:17.124260 14717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:17.124552 14717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:17.124598 14717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:17.125980 14717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:17.126055 14717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:17.126116 14717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:34:17.127857 14717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37065
--tserver_master_addrs=127.0.98.62:36203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:17.128155 14717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:17.128433 14717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:17.134477 14743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:17.134440 14742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:17.134621 14745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:17.135068 14717 server_base.cc:1034] running on GCE node
I20260618 23:34:17.135329 14717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:17.135696 14717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:17.136906 14717 hybrid_clock.cc:648] HybridClock initialized: now 1781825657136881 us; error 47 us; skew 500 ppm
Jun 18 23:34:17 dist-test-slave-6224 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:17.140096 14717 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:34:17.140709 14717 webserver.cc:460] Webserver started at http://127.0.98.1:37065/ using document root <none> and password file <none>
I20260618 23:34:17.140966 14717 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:17.141062 14717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:17.142774 14717 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:17.143632 14752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:17.143890 14717 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:17.144042 14717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "efa7e30dcd9841d993d31ba6b984fd0a"
format_stamp: "Formatted at 2026-06-18 23:34:16 on dist-test-slave-6224"
server_key: "59b3186c3534a30250b8c69852037fa1"
server_key_iv: "80c5e5339cff7b370a5b003e7cf66109"
server_key_version: "encryptionkey@0"
I20260618 23:34:17.144615 14717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:17.162714 14717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:17.165522 14717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:17.165695 14717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:17.165983 14717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:34:17.166602 14759 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260618 23:34:17.168491 14717 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 23:34:17.168570 14717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260618 23:34:17.168605 14717 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 23:34:17.169660 14717 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 23:34:17.169714 14717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260618 23:34:17.169867 14759 tablet_bootstrap.cc:492] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a: Bootstrap starting.
I20260618 23:34:17.177676 14717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44499
I20260618 23:34:17.178043 14866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44499 every 8 connection(s)
I20260618 23:34:17.178193 14717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:34:17.184314 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 14717
I20260618 23:34:17.184397 14759 log.cc:824] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:17.184708 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39423
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36203
--webserver_port=36367
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:34:17.188058 14759 tablet_bootstrap.cc:492] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:34:17.188462 14759 tablet_bootstrap.cc:492] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a: Bootstrap complete.
I20260618 23:34:17.190172 14759 ts_tablet_manager.cc:1397] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a: Time spent bootstrapping tablet: real 0.020s user 0.015s sys 0.004s
Jun 18 23:34:17 dist-test-slave-6224 krb5kdc[14099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:34:17.192291 14759 raft_consensus.cc:348] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } }
I20260618 23:34:17.192783 14759 raft_consensus.cc:729] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa7e30dcd9841d993d31ba6b984fd0a, State: Initialized, Role: FOLLOWER
I20260618 23:34:17.193439 14759 consensus_queue.cc:260] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } }
I20260618 23:34:17.194200 14759 ts_tablet_manager.cc:1428] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260618 23:34:17.196776 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:17.179668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51823 (local address 127.0.98.62:36203)
0618 23:34:17.179959 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:17.179970 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:17.180022 (+ 52us) server_negotiation.cc:407] Connection header received
0618 23:34:17.180890 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:17.180898 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:17.180965 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:17.181133 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:17.182465 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.183931 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.188312 (+ 4381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.188940 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.189075 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:17.192510 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:17.192514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:17.192515 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:17.192526 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:17.194698 (+ 2172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:17.195264 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:17.195268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:17.195268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:17.195386 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:17.195688 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:17.195689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:17.195690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:17.195980 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:17.196051 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:17.196451 (+ 400us) server_negotiation.cc:299] Negotiation successful
0618 23:34:17.196605 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":143,"threads_started":1}
I20260618 23:34:17.197290 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:17.179174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.1:51823)
0618 23:34:17.179443 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:17.179522 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:17.180611 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:17.181258 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:17.181268 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:17.181852 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:17.182277 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:17.182289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.186733 (+ 4444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.186734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:17.188170 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:17.188177 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.189296 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.189299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:17.189463 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:17.190219 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:17.190230 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:17.192375 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:17.194875 (+ 2500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:17.194881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:17.194884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:17.195167 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:17.195460 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:17.195461 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:17.195461 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:17.195610 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:17.196091 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:17.196097 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:17.196249 (+ 152us) client_negotiation.cc:769] Sending connection context
0618 23:34:17.196913 (+ 664us) client_negotiation.cc:240] Negotiation successful
0618 23:34:17.197106 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":112,"threads_started":1}
I20260618 23:34:17.198318 14867 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36203
I20260618 23:34:17.198436 14867 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:17.198617 14867 heartbeater.cc:511] Master 127.0.98.62:36203 requested a full tablet report, sending...
I20260618 23:34:17.199633 14679 ts_manager.cc:194] Registered new tserver with Master: efa7e30dcd9841d993d31ba6b984fd0a (127.0.98.1:44499)
I20260618 23:34:17.201032 14679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51823
I20260618 23:34:17.202162 14867 heartbeater.cc:503] Master 127.0.98.62:36203 was elected leader, sending a full tablet report...
Jun 18 23:34:17 dist-test-slave-6224 krb5kdc[14099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260618 23:34:17.208756 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:17.200708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54306 (local address 127.0.98.1:44499)
0618 23:34:17.200904 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:17.200909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:17.200918 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:34:17.200957 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:17.200959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:17.200973 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:17.201056 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:17.201535 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.202406 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.203353 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.203727 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.203809 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:17.205854 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:17.205861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:17.205861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:17.205868 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:17.207442 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:17.207886 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:17.207889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:17.207890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:17.207950 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:17.208145 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:17.208146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:17.208146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:17.208296 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:17.208385 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:17.208566 (+ 181us) server_negotiation.cc:299] Negotiation successful
0618 23:34:17.208653 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":165,"thread_start_us":137,"threads_started":1}
I20260618 23:34:17.208765 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:17.200396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44499 (local address 127.0.0.1:54306)
0618 23:34:17.200578 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:17.200612 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:17.200680 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:17.201064 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:17.201069 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:17.201398 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:17.201469 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:17.201471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.202490 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.202491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:17.203217 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:17.203220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.203808 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.203811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:17.203888 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:17.204472 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:17.204477 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:17.205752 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:17.207567 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:17.207574 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:17.207575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:17.207782 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:17.208019 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:17.208020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:17.208021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:17.208081 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:17.208378 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:17.208384 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:17.208435 (+ 51us) client_negotiation.cc:769] Sending connection context
0618 23:34:17.208529 (+ 94us) client_negotiation.cc:240] Negotiation successful
0618 23:34:17.208652 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":121,"threads_started":1}
I20260618 23:34:17.209290 14788 tablet_service.cc:1514] Processing DeleteTablet for tablet 7178ed09ddda4c8fb36da19a703e2773 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:34:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54306
I20260618 23:34:17.210057 14876 tablet_replica.cc:330] stopping tablet replica
I20260618 23:34:17.210203 14876 raft_consensus.cc:2229] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:34:17.210301 14876 raft_consensus.cc:2258] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:34:17.210690 14876 ts_tablet_manager.cc:1905] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:34:17.212844 14876 ts_tablet_manager.cc:1918] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:34:17.212937 14876 log.cc:1196] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/wal/wals/7178ed09ddda4c8fb36da19a703e2773
I20260618 23:34:17.213173 14876 ts_tablet_manager.cc:1939] T 7178ed09ddda4c8fb36da19a703e2773 P efa7e30dcd9841d993d31ba6b984fd0a: Deleting consensus metadata
I20260618 23:34:17.213747 14665 catalog_manager.cc:4841] TS efa7e30dcd9841d993d31ba6b984fd0a (127.0.98.1:44499): tablet 7178ed09ddda4c8fb36da19a703e2773 (table test-table [id=d4781e5320074ec7bd046facae9f7b6f]) successfully deleted
W20260618 23:34:17.284931 14872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:17.285202 14872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:17.285233 14872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:17.286566 14872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:17.286610 14872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:17.286643 14872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:34:17.288245 14872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36367
--tserver_master_addrs=127.0.98.62:36203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:17.288496 14872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:17.288733 14872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:34:17.294700 14872 server_base.cc:1034] running on GCE node
W20260618 23:34:17.294657 14884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:17.294657 14881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:17.294644 14882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:17.295143 14872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:17.295487 14872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:17.296684 14872 hybrid_clock.cc:648] HybridClock initialized: now 1781825657296657 us; error 52 us; skew 500 ppm
Jun 18 23:34:17 dist-test-slave-6224 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:17.299376 14872 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:34:17.299875 14872 webserver.cc:460] Webserver started at http://127.0.98.2:36367/ using document root <none> and password file <none>
I20260618 23:34:17.300078 14872 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:17.300137 14872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:17.301774 14872 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:17.302587 14891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:17.302852 14872 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:17.302991 14872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "be6c382c0e6f45a5b00483b7291bbd19"
format_stamp: "Formatted at 2026-06-18 23:34:16 on dist-test-slave-6224"
server_key: "e10b1e9723185edcf446c300ea3989bf"
server_key_iv: "6d1ca22207a2f1b82b6e5b47465b7eaa"
server_key_version: "encryptionkey@0"
I20260618 23:34:17.303414 14872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:17.323316 14872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:17.326032 14872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:17.326227 14872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:17.326555 14872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:34:17.326922 14872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:34:17.326983 14872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:17.327023 14872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:34:17.327055 14872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:17.334686 14872 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39423
I20260618 23:34:17.334751 15004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39423 every 8 connection(s)
I20260618 23:34:17.335126 14872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
Jun 18 23:34:17 dist-test-slave-6224 krb5kdc[14099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:34:17.342685 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 14872
I20260618 23:34:17.342989 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45521
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36203
--webserver_port=45423
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:34:17.345943 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:17.336087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55415 (local address 127.0.98.62:36203)
0618 23:34:17.336172 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:17.336175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:17.336398 (+ 223us) server_negotiation.cc:407] Connection header received
0618 23:34:17.336898 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:17.336899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:17.336916 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:17.337010 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:17.338043 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.338614 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.339845 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.340081 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.340157 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:17.342417 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:17.342418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:17.342419 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:17.342426 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:17.344232 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:17.344749 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:17.344751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:17.344751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:17.344815 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:17.345118 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:17.345119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:17.345119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:17.345263 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:17.345375 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:17.345772 (+ 397us) server_negotiation.cc:299] Negotiation successful
0618 23:34:17.345862 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 23:34:17.346449 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:17.336047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.2:55415)
0618 23:34:17.336216 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:17.336262 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:17.336799 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:17.337082 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:17.337085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:17.337704 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:17.337941 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:17.337946 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.338744 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.338745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:17.339731 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:17.339734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.340182 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.340183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:17.340291 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:17.340896 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:17.340905 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:17.342311 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:17.344352 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:17.344366 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:17.344370 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:17.344664 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:17.344903 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:17.344904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:17.344904 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:17.345044 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:17.345395 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:17.345403 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:17.345598 (+ 195us) client_negotiation.cc:769] Sending connection context
0618 23:34:17.345747 (+ 149us) client_negotiation.cc:240] Negotiation successful
0618 23:34:17.345955 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260618 23:34:17.347052 15005 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36203
I20260618 23:34:17.347198 15005 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:17.347437 15005 heartbeater.cc:511] Master 127.0.98.62:36203 requested a full tablet report, sending...
I20260618 23:34:17.347988 14679 ts_manager.cc:194] Registered new tserver with Master: be6c382c0e6f45a5b00483b7291bbd19 (127.0.98.2:39423)
I20260618 23:34:17.348433 14679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55415
W20260618 23:34:17.430219 15008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:17.430470 15008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:17.430509 15008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:17.431672 15008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:17.431743 15008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:17.431797 15008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:34:17.433300 15008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45423
--tserver_master_addrs=127.0.98.62:36203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:17.433574 15008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:17.433815 15008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:17.439209 15013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:17.439234 15008 server_base.cc:1034] running on GCE node
W20260618 23:34:17.439234 15014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:17.439209 15016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:17.439862 15008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:17.440142 15008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:17.441310 15008 hybrid_clock.cc:648] HybridClock initialized: now 1781825657441290 us; error 36 us; skew 500 ppm
Jun 18 23:34:17 dist-test-slave-6224 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:17.443862 15008 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:34:17.444319 15008 webserver.cc:460] Webserver started at http://127.0.98.3:45423/ using document root <none> and password file <none>
I20260618 23:34:17.444555 15008 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:17.444617 15008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:17.446295 15008 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:17.447079 15023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:17.447304 15008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:17.447417 15008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "0ebf2f83a0264e7cb5257abb7942a1ec"
format_stamp: "Formatted at 2026-06-18 23:34:16 on dist-test-slave-6224"
server_key: "24fdfc0998be3058d2eaca70e24d66f0"
server_key_iv: "261bd1ef601f4d73ef97c4d43442d216"
server_key_version: "encryptionkey@0"
I20260618 23:34:17.447928 15008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:17.479671 15008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:17.482278 15008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:17.482411 15008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:17.482642 15008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:34:17.483023 15008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:34:17.483063 15008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:17.483089 15008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:34:17.483103 15008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:17.490644 15008 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45521
I20260618 23:34:17.490741 15136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45521 every 8 connection(s)
I20260618 23:34:17.491163 15008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
Jun 18 23:34:17 dist-test-slave-6224 krb5kdc[14099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:34:17.499763 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 15008
I20260618 23:34:17.502178 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:17.492130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36149 (local address 127.0.98.62:36203)
0618 23:34:17.492224 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:17.492226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:17.492431 (+ 205us) server_negotiation.cc:407] Connection header received
0618 23:34:17.492940 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:17.492941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:17.492951 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:17.493034 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:17.494041 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.494578 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.495861 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.496204 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.496299 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:17.498943 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:17.498946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:17.498946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:17.498955 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:17.500655 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:17.501062 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:17.501063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:17.501064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:17.501133 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:17.501402 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:17.501403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:17.501404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:17.501570 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:17.501648 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:17.502026 (+ 378us) server_negotiation.cc:299] Negotiation successful
0618 23:34:17.502110 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 23:34:17.502456 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:17.492095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.3:36149)
0618 23:34:17.492282 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:17.492325 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:17.492839 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:17.493111 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:17.493114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:17.493723 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:17.493935 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:17.493940 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.494666 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.494667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:17.495734 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:17.495741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.496402 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.496404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:17.496585 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:17.497243 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:17.497254 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:17.498824 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:17.500735 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:17.500742 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:17.500745 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:17.500979 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:17.501215 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:17.501216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:17.501216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:17.501328 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:17.501650 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:17.501656 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:17.501843 (+ 187us) client_negotiation.cc:769] Sending connection context
0618 23:34:17.501973 (+ 130us) client_negotiation.cc:240] Negotiation successful
0618 23:34:17.502169 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":110,"threads_started":1}
I20260618 23:34:17.503072 15137 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36203
I20260618 23:34:17.503213 15137 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:17.503438 15137 heartbeater.cc:511] Master 127.0.98.62:36203 requested a full tablet report, sending...
I20260618 23:34:17.503986 14679 ts_manager.cc:194] Registered new tserver with Master: 0ebf2f83a0264e7cb5257abb7942a1ec (127.0.98.3:45521)
I20260618 23:34:17.504451 14679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36149
I20260618 23:34:17.508273 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:17.500275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.0.98.62:36203)
0618 23:34:17.500761 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:17.500763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:17.500768 (+ 5us) server_negotiation.cc:407] Connection header received
0618 23:34:17.500784 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:17.500784 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:17.500795 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:17.500871 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:17.501641 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.502605 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.503708 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.504005 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.504104 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:17.505442 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:17.505444 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:17.505444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:17.505452 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:17.507121 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:17.507523 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:17.507525 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:17.507525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:17.507611 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:17.507852 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:17.507853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:17.507853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:17.507960 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:17.508105 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:17.508137 (+ 32us) server_negotiation.cc:299] Negotiation successful
0618 23:34:17.508202 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":76,"threads_started":1}
I20260618 23:34:17.509092 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:34:17.512049 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:17.509449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40268 (local address 127.0.98.62:36203)
0618 23:34:17.509509 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:17.509511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:17.509651 (+ 140us) server_negotiation.cc:407] Connection header received
0618 23:34:17.509739 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:17.509740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:17.509770 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:17.509846 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:34:17.509985 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.510463 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.511234 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.511428 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.511505 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:17.511738 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:34:17.511832 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:34:17.511928 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:17.511946 (+ 18us) server_negotiation.cc:299] Negotiation successful
0618 23:34:17.511970 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260618 23:34:17.512375 14679 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40268:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:34:17.514091 14679 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:34:17.516706 14788 tablet_service.cc:1467] Processing CreateTablet for tablet 633f57b3df6c4c5ea74c93ee2c9696b8 (DEFAULT_TABLE table=test-table [id=f92cc74933194c388cf97728a99943a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:34:17.516996 14788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 633f57b3df6c4c5ea74c93ee2c9696b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:34:17.519186 14759 tablet_bootstrap.cc:492] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a: Bootstrap starting.
I20260618 23:34:17.519690 14759 tablet_bootstrap.cc:654] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a: Neither blocks nor log segments found. Creating new log.
I20260618 23:34:17.520443 14759 tablet_bootstrap.cc:492] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a: No bootstrap required, opened a new log
I20260618 23:34:17.520536 14759 ts_tablet_manager.cc:1397] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260618 23:34:17.520638 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:17.516254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39423 (local address 127.0.0.1:52318)
0618 23:34:17.516324 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:17.516339 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:17.516395 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:17.516943 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:17.516944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:17.516946 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:34:17.516991 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:17.516993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.518102 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.518103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:17.519025 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:17.519027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.520334 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.520335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:17.520405 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:17.520433 (+ 28us) client_negotiation.cc:769] Sending connection context
0618 23:34:17.520467 (+ 34us) client_negotiation.cc:240] Negotiation successful
0618 23:34:17.520566 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260618 23:34:17.521080 15143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:17.516262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45521 (local address 127.0.0.1:51262)
0618 23:34:17.516445 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:17.516459 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:17.516518 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:17.517373 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:17.517374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:17.517377 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:34:17.517434 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:17.517436 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.518774 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.518775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:17.519861 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:17.519863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.520837 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.520839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:17.520927 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:17.520935 (+ 8us) client_negotiation.cc:769] Sending connection context
0618 23:34:17.520967 (+ 32us) client_negotiation.cc:240] Negotiation successful
0618 23:34:17.521013 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":21,"thread_start_us":97,"threads_started":1}
I20260618 23:34:17.520987 14759 raft_consensus.cc:348] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } }
I20260618 23:34:17.521173 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:17.516484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52318 (local address 127.0.98.2:39423)
0618 23:34:17.516720 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:17.516727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:17.516740 (+ 13us) server_negotiation.cc:407] Connection header received
0618 23:34:17.516790 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:17.516792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:17.516810 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:17.516911 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:34:17.517134 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.517973 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.519381 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.520236 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.520933 (+ 697us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:17.521006 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:17.521037 (+ 31us) server_negotiation.cc:299] Negotiation successful
0618 23:34:17.521080 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":120,"threads_started":1}
I20260618 23:34:17.521145 14759 raft_consensus.cc:374] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:34:17.521219 14759 raft_consensus.cc:729] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa7e30dcd9841d993d31ba6b984fd0a, State: Initialized, Role: FOLLOWER
I20260618 23:34:17.521283 14759 consensus_queue.cc:260] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } }
I20260618 23:34:17.521385 14759 ts_tablet_manager.cc:1428] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260618 23:34:17.521561 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:17.516483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51262 (local address 127.0.98.3:45521)
0618 23:34:17.517171 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:17.517175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:17.517186 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:34:17.517232 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:17.517234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:17.517260 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:17.517502 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:34:17.517582 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.518641 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.519963 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.520708 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.521301 (+ 593us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:17.521384 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:17.521412 (+ 28us) server_negotiation.cc:299] Negotiation successful
0618 23:34:17.521465 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":124,"threads_started":1}
I20260618 23:34:17.522192 14939 tablet_service.cc:1467] Processing CreateTablet for tablet 633f57b3df6c4c5ea74c93ee2c9696b8 (DEFAULT_TABLE table=test-table [id=f92cc74933194c388cf97728a99943a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:34:17.522264 15071 tablet_service.cc:1467] Processing CreateTablet for tablet 633f57b3df6c4c5ea74c93ee2c9696b8 (DEFAULT_TABLE table=test-table [id=f92cc74933194c388cf97728a99943a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:34:17.522617 14939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 633f57b3df6c4c5ea74c93ee2c9696b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:34:17.522617 15071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 633f57b3df6c4c5ea74c93ee2c9696b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:34:17.526849 15148 tablet_bootstrap.cc:492] T 633f57b3df6c4c5ea74c93ee2c9696b8 P 0ebf2f83a0264e7cb5257abb7942a1ec: Bootstrap starting.
I20260618 23:34:17.527685 15148 tablet_bootstrap.cc:654] T 633f57b3df6c4c5ea74c93ee2c9696b8 P 0ebf2f83a0264e7cb5257abb7942a1ec: Neither blocks nor log segments found. Creating new log.
I20260618 23:34:17.527760 15147 tablet_bootstrap.cc:492] T 633f57b3df6c4c5ea74c93ee2c9696b8 P be6c382c0e6f45a5b00483b7291bbd19: Bootstrap starting.
I20260618 23:34:17.528277 15148 log.cc:824] T 633f57b3df6c4c5ea74c93ee2c9696b8 P 0ebf2f83a0264e7cb5257abb7942a1ec: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:17.528795 15147 tablet_bootstrap.cc:654] T 633f57b3df6c4c5ea74c93ee2c9696b8 P be6c382c0e6f45a5b00483b7291bbd19: Neither blocks nor log segments found. Creating new log.
I20260618 23:34:17.529270 15148 tablet_bootstrap.cc:492] T 633f57b3df6c4c5ea74c93ee2c9696b8 P 0ebf2f83a0264e7cb5257abb7942a1ec: No bootstrap required, opened a new log
I20260618 23:34:17.529342 15148 ts_tablet_manager.cc:1397] T 633f57b3df6c4c5ea74c93ee2c9696b8 P 0ebf2f83a0264e7cb5257abb7942a1ec: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260618 23:34:17.529438 15147 log.cc:824] T 633f57b3df6c4c5ea74c93ee2c9696b8 P be6c382c0e6f45a5b00483b7291bbd19: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:17.530480 15147 tablet_bootstrap.cc:492] T 633f57b3df6c4c5ea74c93ee2c9696b8 P be6c382c0e6f45a5b00483b7291bbd19: No bootstrap required, opened a new log
I20260618 23:34:17.530571 15147 ts_tablet_manager.cc:1397] T 633f57b3df6c4c5ea74c93ee2c9696b8 P be6c382c0e6f45a5b00483b7291bbd19: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 23:34:17.531159 15148 raft_consensus.cc:348] T 633f57b3df6c4c5ea74c93ee2c9696b8 P 0ebf2f83a0264e7cb5257abb7942a1ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } }
I20260618 23:34:17.531323 15148 raft_consensus.cc:374] T 633f57b3df6c4c5ea74c93ee2c9696b8 P 0ebf2f83a0264e7cb5257abb7942a1ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:34:17.531347 15148 raft_consensus.cc:729] T 633f57b3df6c4c5ea74c93ee2c9696b8 P 0ebf2f83a0264e7cb5257abb7942a1ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ebf2f83a0264e7cb5257abb7942a1ec, State: Initialized, Role: FOLLOWER
I20260618 23:34:17.531497 15148 consensus_queue.cc:260] T 633f57b3df6c4c5ea74c93ee2c9696b8 P 0ebf2f83a0264e7cb5257abb7942a1ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } }
I20260618 23:34:17.531865 15148 ts_tablet_manager.cc:1428] T 633f57b3df6c4c5ea74c93ee2c9696b8 P 0ebf2f83a0264e7cb5257abb7942a1ec: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:34:17.531924 15137 heartbeater.cc:503] Master 127.0.98.62:36203 was elected leader, sending a full tablet report...
I20260618 23:34:17.532754 15147 raft_consensus.cc:348] T 633f57b3df6c4c5ea74c93ee2c9696b8 P be6c382c0e6f45a5b00483b7291bbd19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } }
I20260618 23:34:17.532976 15147 raft_consensus.cc:374] T 633f57b3df6c4c5ea74c93ee2c9696b8 P be6c382c0e6f45a5b00483b7291bbd19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:34:17.533025 15147 raft_consensus.cc:729] T 633f57b3df6c4c5ea74c93ee2c9696b8 P be6c382c0e6f45a5b00483b7291bbd19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be6c382c0e6f45a5b00483b7291bbd19, State: Initialized, Role: FOLLOWER
I20260618 23:34:17.533151 15147 consensus_queue.cc:260] T 633f57b3df6c4c5ea74c93ee2c9696b8 P be6c382c0e6f45a5b00483b7291bbd19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } }
I20260618 23:34:17.533527 15147 ts_tablet_manager.cc:1428] T 633f57b3df6c4c5ea74c93ee2c9696b8 P be6c382c0e6f45a5b00483b7291bbd19: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:34:17.533555 15005 heartbeater.cc:503] Master 127.0.98.62:36203 was elected leader, sending a full tablet report...
I20260618 23:34:17.579980 14873 raft_consensus.cc:482] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:34:17.580194 14873 raft_consensus.cc:504] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } }
I20260618 23:34:17.580665 14873 leader_election.cc:288] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be6c382c0e6f45a5b00483b7291bbd19 (127.0.98.2:39423), 0ebf2f83a0264e7cb5257abb7942a1ec (127.0.98.3:45521)
I20260618 23:34:17.583869 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:17.581043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32857 (local address 127.0.98.3:45521)
0618 23:34:17.581102 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:17.581105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:17.581119 (+ 14us) server_negotiation.cc:407] Connection header received
0618 23:34:17.581231 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:17.581233 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:17.581253 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:17.581334 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:34:17.581502 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.581999 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.583047 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.583443 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.583526 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:17.583538 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:17.583745 (+ 207us) server_negotiation.cc:299] Negotiation successful
0618 23:34:17.583785 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260618 23:34:17.583869 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:17.580909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45521 (local address 127.0.98.1:32857)
0618 23:34:17.580999 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:17.581021 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:17.581097 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:17.581345 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:17.581346 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:17.581348 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:34:17.581426 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:17.581428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.582082 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.582083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:17.582951 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:17.582953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.583525 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.583526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:17.583654 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:17.583671 (+ 17us) client_negotiation.cc:769] Sending connection context
0618 23:34:17.583739 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 23:34:17.583785 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":26}
I20260618 23:34:17.584053 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:17.581120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39713 (local address 127.0.98.2:39423)
0618 23:34:17.581169 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:17.581171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:17.581322 (+ 151us) server_negotiation.cc:407] Connection header received
0618 23:34:17.581401 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:17.581402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:17.581419 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:17.581488 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:34:17.581679 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.582191 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.583206 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.583673 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.583759 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:17.583773 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:17.583927 (+ 154us) server_negotiation.cc:299] Negotiation successful
0618 23:34:17.583967 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:34:17.584079 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:17.581023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39423 (local address 127.0.98.1:39713)
0618 23:34:17.581231 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:17.581248 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:17.581335 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:17.581491 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:17.581492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:17.581494 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:34:17.581604 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:17.581606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.582289 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.582290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:17.583114 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:17.583115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.583778 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.583779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:17.583851 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:17.583856 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:34:17.583922 (+ 66us) client_negotiation.cc:240] Negotiation successful
0618 23:34:17.583965 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":1,"thread_start_us":109,"threads_started":1}
I20260618 23:34:17.584285 15091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "633f57b3df6c4c5ea74c93ee2c9696b8" candidate_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" is_pre_election: true
I20260618 23:34:17.584470 15091 raft_consensus.cc:2454] T 633f57b3df6c4c5ea74c93ee2c9696b8 P 0ebf2f83a0264e7cb5257abb7942a1ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efa7e30dcd9841d993d31ba6b984fd0a in term 0.
I20260618 23:34:17.584682 14959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "633f57b3df6c4c5ea74c93ee2c9696b8" candidate_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be6c382c0e6f45a5b00483b7291bbd19" is_pre_election: true
I20260618 23:34:17.584846 14959 raft_consensus.cc:2454] T 633f57b3df6c4c5ea74c93ee2c9696b8 P be6c382c0e6f45a5b00483b7291bbd19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efa7e30dcd9841d993d31ba6b984fd0a in term 0.
I20260618 23:34:17.584803 14756 leader_election.cc:302] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ebf2f83a0264e7cb5257abb7942a1ec, efa7e30dcd9841d993d31ba6b984fd0a; no voters:
I20260618 23:34:17.585004 14873 raft_consensus.cc:2781] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:34:17.585078 14873 raft_consensus.cc:482] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:34:17.585098 14873 raft_consensus.cc:3037] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 0 FOLLOWER]: Advancing to term 1
W20260618 23:34:17.585716 15006 tablet.cc:2369] T 633f57b3df6c4c5ea74c93ee2c9696b8 P be6c382c0e6f45a5b00483b7291bbd19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:34:17.586077 14873 raft_consensus.cc:504] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } }
I20260618 23:34:17.586269 14873 leader_election.cc:288] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 1 election: Requested vote from peers be6c382c0e6f45a5b00483b7291bbd19 (127.0.98.2:39423), 0ebf2f83a0264e7cb5257abb7942a1ec (127.0.98.3:45521)
I20260618 23:34:17.586431 14959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "633f57b3df6c4c5ea74c93ee2c9696b8" candidate_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be6c382c0e6f45a5b00483b7291bbd19"
I20260618 23:34:17.586496 14959 raft_consensus.cc:3037] T 633f57b3df6c4c5ea74c93ee2c9696b8 P be6c382c0e6f45a5b00483b7291bbd19 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:34:17.586517 15091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "633f57b3df6c4c5ea74c93ee2c9696b8" candidate_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec"
I20260618 23:34:17.586678 15091 raft_consensus.cc:3037] T 633f57b3df6c4c5ea74c93ee2c9696b8 P 0ebf2f83a0264e7cb5257abb7942a1ec [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:34:17.587337 14959 raft_consensus.cc:2454] T 633f57b3df6c4c5ea74c93ee2c9696b8 P be6c382c0e6f45a5b00483b7291bbd19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efa7e30dcd9841d993d31ba6b984fd0a in term 1.
I20260618 23:34:17.587536 14755 leader_election.cc:302] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be6c382c0e6f45a5b00483b7291bbd19, efa7e30dcd9841d993d31ba6b984fd0a; no voters:
I20260618 23:34:17.587662 14873 raft_consensus.cc:2781] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:34:17.587782 15091 raft_consensus.cc:2454] T 633f57b3df6c4c5ea74c93ee2c9696b8 P 0ebf2f83a0264e7cb5257abb7942a1ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efa7e30dcd9841d993d31ba6b984fd0a in term 1.
I20260618 23:34:17.587852 14873 raft_consensus.cc:686] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 LEADER]: Becoming Leader. State: Replica: efa7e30dcd9841d993d31ba6b984fd0a, State: Running, Role: LEADER
I20260618 23:34:17.587939 14873 consensus_queue.cc:237] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } }
I20260618 23:34:17.588752 14679 catalog_manager.cc:5495] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a reported cstate change: term changed from 0 to 1, leader changed from <none> to efa7e30dcd9841d993d31ba6b984fd0a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } health_report { overall_health: HEALTHY } } }
I20260618 23:34:17.596024 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:17.593603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54320 (local address 127.0.98.1:44499)
0618 23:34:17.593652 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:17.593655 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:17.593663 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:34:17.593737 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:17.593740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:17.593761 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:17.593888 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:34:17.593996 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.594481 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.595205 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:17.595414 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:17.595493 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:17.595659 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:34:17.595762 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:34:17.595859 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:17.595909 (+ 50us) server_negotiation.cc:299] Negotiation successful
0618 23:34:17.595933 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260618 23:34:17.597890 15091 raft_consensus.cc:1261] T 633f57b3df6c4c5ea74c93ee2c9696b8 P 0ebf2f83a0264e7cb5257abb7942a1ec [term 1 FOLLOWER]: Refusing update from remote peer efa7e30dcd9841d993d31ba6b984fd0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:34:17.597891 14959 raft_consensus.cc:1261] T 633f57b3df6c4c5ea74c93ee2c9696b8 P be6c382c0e6f45a5b00483b7291bbd19 [term 1 FOLLOWER]: Refusing update from remote peer efa7e30dcd9841d993d31ba6b984fd0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:34:17.598346 14873 consensus_queue.cc:1035] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:34:17.598554 15154 consensus_queue.cc:1035] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:34:17.603574 14679 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:40268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:34:17.603694 14679 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40268:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:34:17.604615 14679 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470: Sending DeleteTablet for 3 replicas of tablet 633f57b3df6c4c5ea74c93ee2c9696b8
I20260618 23:34:17.605065 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 14717
I20260618 23:34:17.605060 14788 tablet_service.cc:1514] Processing DeleteTablet for tablet 633f57b3df6c4c5ea74c93ee2c9696b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:34:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54306
I20260618 23:34:17.605077 14939 tablet_service.cc:1514] Processing DeleteTablet for tablet 633f57b3df6c4c5ea74c93ee2c9696b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:34:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52318
I20260618 23:34:17.605103 15071 tablet_service.cc:1514] Processing DeleteTablet for tablet 633f57b3df6c4c5ea74c93ee2c9696b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:34:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51262
I20260618 23:34:17.605307 14876 tablet_replica.cc:330] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a: stopping tablet replica
I20260618 23:34:17.605389 14876 raft_consensus.cc:2229] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:34:17.605472 14876 raft_consensus.cc:2258] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:34:17.606264 15168 tablet_replica.cc:330] T 633f57b3df6c4c5ea74c93ee2c9696b8 P 0ebf2f83a0264e7cb5257abb7942a1ec: stopping tablet replica
I20260618 23:34:17.606400 15168 raft_consensus.cc:2229] T 633f57b3df6c4c5ea74c93ee2c9696b8 P 0ebf2f83a0264e7cb5257abb7942a1ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:34:17.606520 15168 raft_consensus.cc:2258] T 633f57b3df6c4c5ea74c93ee2c9696b8 P 0ebf2f83a0264e7cb5257abb7942a1ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:34:17.606747 15167 tablet_replica.cc:330] T 633f57b3df6c4c5ea74c93ee2c9696b8 P be6c382c0e6f45a5b00483b7291bbd19: stopping tablet replica
I20260618 23:34:17.606868 15167 raft_consensus.cc:2229] T 633f57b3df6c4c5ea74c93ee2c9696b8 P be6c382c0e6f45a5b00483b7291bbd19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:34:17.606982 15167 raft_consensus.cc:2258] T 633f57b3df6c4c5ea74c93ee2c9696b8 P be6c382c0e6f45a5b00483b7291bbd19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:34:17.607510 15168 ts_tablet_manager.cc:1905] T 633f57b3df6c4c5ea74c93ee2c9696b8 P 0ebf2f83a0264e7cb5257abb7942a1ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:34:17.607741 15167 ts_tablet_manager.cc:1905] T 633f57b3df6c4c5ea74c93ee2c9696b8 P be6c382c0e6f45a5b00483b7291bbd19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:34:17.609543 15167 ts_tablet_manager.cc:1918] T 633f57b3df6c4c5ea74c93ee2c9696b8 P be6c382c0e6f45a5b00483b7291bbd19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:34:17.609647 15167 log.cc:1196] T 633f57b3df6c4c5ea74c93ee2c9696b8 P be6c382c0e6f45a5b00483b7291bbd19: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/633f57b3df6c4c5ea74c93ee2c9696b8
I20260618 23:34:17.609848 15168 ts_tablet_manager.cc:1918] T 633f57b3df6c4c5ea74c93ee2c9696b8 P 0ebf2f83a0264e7cb5257abb7942a1ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:34:17.609941 15168 log.cc:1196] T 633f57b3df6c4c5ea74c93ee2c9696b8 P 0ebf2f83a0264e7cb5257abb7942a1ec: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/633f57b3df6c4c5ea74c93ee2c9696b8
I20260618 23:34:17.610056 15167 ts_tablet_manager.cc:1939] T 633f57b3df6c4c5ea74c93ee2c9696b8 P be6c382c0e6f45a5b00483b7291bbd19: Deleting consensus metadata
I20260618 23:34:17.610297 15168 ts_tablet_manager.cc:1939] T 633f57b3df6c4c5ea74c93ee2c9696b8 P 0ebf2f83a0264e7cb5257abb7942a1ec: Deleting consensus metadata
I20260618 23:34:17.610632 14665 catalog_manager.cc:4841] TS be6c382c0e6f45a5b00483b7291bbd19 (127.0.98.2:39423): tablet 633f57b3df6c4c5ea74c93ee2c9696b8 (table test-table [id=f92cc74933194c388cf97728a99943a8]) successfully deleted
I20260618 23:34:17.610776 14666 catalog_manager.cc:4841] TS 0ebf2f83a0264e7cb5257abb7942a1ec (127.0.98.3:45521): tablet 633f57b3df6c4c5ea74c93ee2c9696b8 (table test-table [id=f92cc74933194c388cf97728a99943a8]) successfully deleted
W20260618 23:34:17.612622 14665 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44499 (error 108)
I20260618 23:34:17.612987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 14872
I20260618 23:34:17.613193 15143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:17.613061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44499 (local address 127.0.0.1:54324)
0618 23:34:17.613126 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:17.613160 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44499: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260618 23:34:17.614127 14665 catalog_manager.cc:4584] TS efa7e30dcd9841d993d31ba6b984fd0a (127.0.98.1:44499): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 633f57b3df6c4c5ea74c93ee2c9696b8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44499: connect: Connection refused (error 111)
I20260618 23:34:17.619540 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 15008
I20260618 23:34:17.624789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 14646
I20260618 23:34:17.631536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36203
--webserver_interface=127.0.98.62
--webserver_port=34315
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36203
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:34:17.730963 15170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:17.731240 15170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:17.731266 15170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:17.732640 15170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:34:17.732699 15170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:17.732714 15170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:17.732728 15170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:34:17.732769 15170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:34:17.734414 15170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36203
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:17.734730 15170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:17.735029 15170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:17.740823 15176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:17.740774 15175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:17.740760 15178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:17.741101 15170 server_base.cc:1034] running on GCE node
I20260618 23:34:17.741418 15170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:17.741766 15170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:17.742947 15170 hybrid_clock.cc:648] HybridClock initialized: now 1781825657742919 us; error 57 us; skew 500 ppm
Jun 18 23:34:17 dist-test-slave-6224 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:17.745623 15170 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:34:17.746083 15170 webserver.cc:460] Webserver started at http://127.0.98.62:34315/ using document root <none> and password file <none>
I20260618 23:34:17.746295 15170 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:17.746352 15170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:17.748052 15170 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:34:17.748867 15185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:17.749135 15170 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:17.749260 15170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "aa04e35ddeee494b89559a1f529a9470"
format_stamp: "Formatted at 2026-06-18 23:34:16 on dist-test-slave-6224"
server_key: "74b633b26516b7b6d0930fd738edf1e3"
server_key_iv: "77ea7ea6e14d5c248d61b55b4b8981a3"
server_key_version: "encryptionkey@0"
I20260618 23:34:17.749742 15170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:17.772387 15170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:17.775027 15170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:17.775159 15170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:17.780159 15170 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36203
I20260618 23:34:17.780251 15237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36203 every 8 connection(s)
I20260618 23:34:17.780707 15170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:34:17.782440 15238 sys_catalog.cc:263] Verifying existing consensus state
I20260618 23:34:17.783177 15238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470: Bootstrap starting.
I20260618 23:34:17.787942 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 15170
I20260618 23:34:17.788286 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44499
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:36203
--webserver_port=37065
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:34:17.792352 15238 log.cc:824] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:17.794580 15238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:34:17.794915 15238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470: Bootstrap complete.
I20260618 23:34:17.797262 15238 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa04e35ddeee494b89559a1f529a9470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } }
I20260618 23:34:17.797493 15238 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa04e35ddeee494b89559a1f529a9470, State: Initialized, Role: FOLLOWER
I20260618 23:34:17.797659 15238 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa04e35ddeee494b89559a1f529a9470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } }
I20260618 23:34:17.797755 15238 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:34:17.797842 15238 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:34:17.797915 15238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 2 FOLLOWER]: Advancing to term 3
I20260618 23:34:17.798938 15238 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa04e35ddeee494b89559a1f529a9470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } }
I20260618 23:34:17.799059 15238 leader_election.cc:302] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa04e35ddeee494b89559a1f529a9470; no voters:
I20260618 23:34:17.799268 15238 leader_election.cc:288] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260618 23:34:17.799304 15242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 3 FOLLOWER]: Leader election won for term 3
I20260618 23:34:17.799432 15242 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [term 3 LEADER]: Becoming Leader. State: Replica: aa04e35ddeee494b89559a1f529a9470, State: Running, Role: LEADER
I20260618 23:34:17.799542 15242 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa04e35ddeee494b89559a1f529a9470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } }
I20260618 23:34:17.799561 15238 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:34:17.799896 15243 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "aa04e35ddeee494b89559a1f529a9470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa04e35ddeee494b89559a1f529a9470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } } }
I20260618 23:34:17.800046 15243 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [sys.catalog]: This master's current role is: LEADER
I20260618 23:34:17.800243 15244 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa04e35ddeee494b89559a1f529a9470. Latest consensus state: current_term: 3 leader_uuid: "aa04e35ddeee494b89559a1f529a9470" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa04e35ddeee494b89559a1f529a9470" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36203 } } }
I20260618 23:34:17.800307 15244 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa04e35ddeee494b89559a1f529a9470 [sys.catalog]: This master's current role is: LEADER
I20260618 23:34:17.800437 15246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:34:17.801445 15246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:34:17.801616 15246 catalog_manager.cc:1259] Loaded cluster ID: 2de2c26b95d0421f8679603ec886707a
I20260618 23:34:17.801964 15246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260618 23:34:17.802235 15246 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8016e2b9ad google::LogMessage::Fail() at ??:0
@ 0x7f8016e2e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f8016e2b4de google::LogMessage::Flush() at ??:0
@ 0x7f8016e2ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f801ed4c0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f801ed558b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8018cf90d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8018cee954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8019b526db start_thread at ??:0
@ 0x7f801592961f clone at ??:0
W20260618 23:34:17.897729 15240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:17.898000 15240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:17.898027 15240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:17.899329 15240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:17.899377 15240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:17.899407 15240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:34:17.901036 15240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44499
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37065
--tserver_master_addrs=127.0.98.62:36203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:17.901336 15240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:17.901611 15240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:17.907284 15277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:17.907249 15274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:17.907289 15275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:17.907882 15240 server_base.cc:1034] running on GCE node
I20260618 23:34:17.908191 15240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:17.908644 15240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:17.909868 15240 hybrid_clock.cc:648] HybridClock initialized: now 1781825657909832 us; error 52 us; skew 500 ppm
Jun 18 23:34:17 dist-test-slave-6224 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:17.913035 15240 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:34:17.913636 15240 webserver.cc:460] Webserver started at http://127.0.98.1:37065/ using document root <none> and password file <none>
I20260618 23:34:17.913923 15240 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:17.913997 15240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:17.915669 15240 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:34:17.916497 15284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:17.916711 15240 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:17.916834 15240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "efa7e30dcd9841d993d31ba6b984fd0a"
format_stamp: "Formatted at 2026-06-18 23:34:16 on dist-test-slave-6224"
server_key: "59b3186c3534a30250b8c69852037fa1"
server_key_iv: "80c5e5339cff7b370a5b003e7cf66109"
server_key_version: "encryptionkey@0"
I20260618 23:34:17.917271 15240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:17.927516 15240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:17.930070 15240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:17.930235 15240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:17.930507 15240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:34:17.931033 15291 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260618 23:34:17.933110 15240 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 23:34:17.933179 15240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260618 23:34:17.933245 15240 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 23:34:17.934024 15240 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 23:34:17.934087 15240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260618 23:34:17.934175 15291 tablet_bootstrap.cc:492] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a: Bootstrap starting.
I20260618 23:34:17.942620 15240 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44499
I20260618 23:34:17.943075 15240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:34:17.943722 15398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44499 every 8 connection(s)
I20260618 23:34:17.944952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 15240
I20260618 23:34:17.945264 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39423
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:36203
--webserver_port=36367
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:34:17.945303 15291 log.cc:824] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:17.948063 15291 tablet_bootstrap.cc:492] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:34:17.948400 15291 tablet_bootstrap.cc:492] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a: Bootstrap complete.
I20260618 23:34:17.949640 15291 ts_tablet_manager.cc:1397] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.003s
I20260618 23:34:17.951032 15291 raft_consensus.cc:348] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } }
I20260618 23:34:17.951366 15291 raft_consensus.cc:729] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: efa7e30dcd9841d993d31ba6b984fd0a, State: Initialized, Role: FOLLOWER
I20260618 23:34:17.951503 15291 consensus_queue.cc:260] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } }
I20260618 23:34:17.951833 15291 ts_tablet_manager.cc:1428] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260618 23:34:18.005038 15401 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:34:17.943907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.1:39879)
0618 23:34:17.944120 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:17.944203 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:17.945003 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:18.004784 (+ 59781us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":81,"threads_started":1}
W20260618 23:34:18.005327 15399 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36203 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36203: Client connection negotiation failed: client connection to 127.0.98.62:36203: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260618 23:34:18.005690 15401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:18.005570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.1:55499)
0618 23:34:18.005625 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:18.005661 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260618 23:34:18.006215 15401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:18.006103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.1:59909)
0618 23:34:18.006171 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:18.006200 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260618 23:34:18.006417 15399 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 23:34:18.039227 15403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:18.039467 15403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:18.039494 15403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:18.040783 15403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:18.040831 15403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:18.040859 15403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:34:18.042397 15403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39423
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36367
--tserver_master_addrs=127.0.98.62:36203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:18.042675 15403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:18.042949 15403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:18.048745 15412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:18.048728 15409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:18.048730 15410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:18.049077 15403 server_base.cc:1034] running on GCE node
I20260618 23:34:18.049369 15403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:18.049656 15403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:18.050815 15403 hybrid_clock.cc:648] HybridClock initialized: now 1781825658050791 us; error 38 us; skew 500 ppm
Jun 18 23:34:18 dist-test-slave-6224 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:18.053547 15403 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:34:18.053983 15403 webserver.cc:460] Webserver started at http://127.0.98.2:36367/ using document root <none> and password file <none>
I20260618 23:34:18.054256 15403 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:18.054330 15403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:18.055830 15403 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:18.056532 15419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:18.056746 15403 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:18.056861 15403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "be6c382c0e6f45a5b00483b7291bbd19"
format_stamp: "Formatted at 2026-06-18 23:34:16 on dist-test-slave-6224"
server_key: "e10b1e9723185edcf446c300ea3989bf"
server_key_iv: "6d1ca22207a2f1b82b6e5b47465b7eaa"
server_key_version: "encryptionkey@0"
I20260618 23:34:18.057264 15403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:18.074411 15403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:18.076653 15403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:18.076828 15403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:18.077107 15403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:34:18.077441 15403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:34:18.077502 15403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:18.077545 15403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:34:18.077593 15403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:18.084408 15403 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39423
I20260618 23:34:18.084525 15532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39423 every 8 connection(s)
I20260618 23:34:18.084870 15403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:34:18.086278 15535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:18.085776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.2:58543)
0618 23:34:18.085978 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:18.086097 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":78,"threads_started":1}
W20260618 23:34:18.086761 15533 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36203 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36203: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
I20260618 23:34:18.087110 15535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:18.086997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.2:53649)
0618 23:34:18.087057 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:18.087090 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260618 23:34:18.087395 15535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:18.087307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.2:44875)
0618 23:34:18.087350 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:18.087372 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260618 23:34:18.087495 15533 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260618 23:34:18.092163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 15403
I20260618 23:34:18.092487 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45521
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:36203
--webserver_port=45423
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:34:18.184305 15536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:18.184593 15536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:18.184636 15536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:18.185909 15536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:18.185990 15536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:18.186044 15536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:34:18.187820 15536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45423
--tserver_master_addrs=127.0.98.62:36203
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:18.188136 15536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:18.188413 15536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:18.194280 15542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:18.194272 15541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:18.194728 15536 server_base.cc:1034] running on GCE node
W20260618 23:34:18.194288 15544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:18.195426 15536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:18.195791 15536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:18.196995 15536 hybrid_clock.cc:648] HybridClock initialized: now 1781825658196964 us; error 50 us; skew 500 ppm
Jun 18 23:34:18 dist-test-slave-6224 krb5kdc[14099](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825658, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:18.200081 15536 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:34:18.200640 15536 webserver.cc:460] Webserver started at http://127.0.98.3:45423/ using document root <none> and password file <none>
I20260618 23:34:18.200930 15536 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:18.201002 15536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:18.202661 15536 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:18.203392 15551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:18.203758 15536 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:18.203867 15536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "0ebf2f83a0264e7cb5257abb7942a1ec"
format_stamp: "Formatted at 2026-06-18 23:34:16 on dist-test-slave-6224"
server_key: "24fdfc0998be3058d2eaca70e24d66f0"
server_key_iv: "261bd1ef601f4d73ef97c4d43442d216"
server_key_version: "encryptionkey@0"
I20260618 23:34:18.204319 15536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:18.218256 15536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:18.220927 15536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:18.221128 15536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:18.221412 15536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:34:18.221786 15536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:34:18.221841 15536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:18.221885 15536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:34:18.221915 15536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:18.229224 15536 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45521
I20260618 23:34:18.229276 15664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45521 every 8 connection(s)
I20260618 23:34:18.229664 15536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:34:18.231004 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:18.230498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.3:47125)
0618 23:34:18.230698 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:18.230797 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":93,"threads_started":1}
W20260618 23:34:18.231491 15665 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:36203 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:36203: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
I20260618 23:34:18.231861 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:18.231748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.3:43755)
0618 23:34:18.231801 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:18.231829 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260618 23:34:18.232165 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:18.232075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.3:53927)
0618 23:34:18.232121 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:18.232145 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260618 23:34:18.232293 15665 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260618 23:34:18.238986 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 15536
I20260618 23:34:19.007290 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:19.006954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.1:50597)
0618 23:34:19.007124 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:19.007206 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":84,"threads_started":1}
I20260618 23:34:19.088224 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:19.087959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.2:42099)
0618 23:34:19.088127 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:19.088172 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":74,"threads_started":1}
I20260618 23:34:19.167706 15671 raft_consensus.cc:482] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:34:19.167939 15671 raft_consensus.cc:504] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } }
I20260618 23:34:19.168370 15671 leader_election.cc:288] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be6c382c0e6f45a5b00483b7291bbd19 (127.0.98.2:39423), 0ebf2f83a0264e7cb5257abb7942a1ec (127.0.98.3:45521)
Jun 18 23:34:19 dist-test-slave-6224 krb5kdc[14099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 18 23:34:19 dist-test-slave-6224 krb5kdc[14099](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260618 23:34:19.180898 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:19.168597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39423 (local address 127.0.98.1:39923)
0618 23:34:19.168771 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:19.168780 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:19.168825 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:19.169980 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:19.169982 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:19.170449 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:19.170680 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:19.170687 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:19.171895 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:19.171897 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:19.172797 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:19.172799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:19.173464 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:19.173466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:19.173578 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:19.174375 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:19.174381 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:19.176384 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:19.179034 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:19.179044 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:19.179047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:19.179385 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:19.179722 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:19.179724 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:19.179724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:19.179903 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:19.180323 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:19.180330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:19.180453 (+ 123us) client_negotiation.cc:769] Sending connection context
0618 23:34:19.180571 (+ 118us) client_negotiation.cc:240] Negotiation successful
0618 23:34:19.180776 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":141,"thread_start_us":67,"threads_started":1}
I20260618 23:34:19.180907 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:19.168967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39923 (local address 127.0.98.2:39423)
0618 23:34:19.169112 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:19.169127 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:19.169169 (+ 42us) server_negotiation.cc:407] Connection header received
0618 23:34:19.169689 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:19.169692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:19.169738 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:19.169926 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:19.170821 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:19.171794 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:19.172954 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:19.173377 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:19.173500 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:19.176506 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:19.176509 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:19.176510 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:19.176521 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:19.178916 (+ 2395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:19.179480 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:19.179482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:19.179483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:19.179629 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:19.179985 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:19.179986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:19.179986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:19.180238 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:19.180339 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:19.180622 (+ 283us) server_negotiation.cc:299] Negotiation successful
0618 23:34:19.180775 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":78,"threads_started":1}
I20260618 23:34:19.181564 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:19.168597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45521 (local address 127.0.98.1:53219)
0618 23:34:19.168646 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:19.168658 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:19.168718 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:19.170003 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:19.170005 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:19.170432 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:19.170681 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:19.170689 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:19.171939 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:19.171941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:19.172796 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:19.172798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:19.173444 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:19.173445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:19.173577 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:19.174243 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:19.174253 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:19.175973 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:19.179483 (+ 3510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:19.179484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:19.179484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:19.179729 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:19.180068 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:19.180070 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:19.180070 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:19.180151 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:19.180656 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:19.180658 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:19.180749 (+ 91us) client_negotiation.cc:769] Sending connection context
0618 23:34:19.181035 (+ 286us) client_negotiation.cc:240] Negotiation successful
0618 23:34:19.181397 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":19}
I20260618 23:34:19.181941 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:19.168774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53219 (local address 127.0.98.3:45521)
0618 23:34:19.168955 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:19.168968 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:19.168994 (+ 26us) server_negotiation.cc:407] Connection header received
0618 23:34:19.169651 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:19.169654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:19.169733 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:19.169944 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:19.170831 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:19.171837 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:19.172903 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:19.173347 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:19.173633 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:19.176149 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:19.176155 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:19.176156 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:19.176511 (+ 355us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:19.178925 (+ 2414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:19.179851 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:19.179860 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:19.179861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:19.179980 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:19.180226 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:19.180227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:19.180228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:19.180548 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:19.180680 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:19.181597 (+ 917us) server_negotiation.cc:299] Negotiation successful
0618 23:34:19.181829 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
W20260618 23:34:19.182081 15287 leader_election.cc:341] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be6c382c0e6f45a5b00483b7291bbd19 (127.0.98.2:39423): Not found: Tablet not found: 633f57b3df6c4c5ea74c93ee2c9696b8
W20260618 23:34:19.182844 15288 leader_election.cc:341] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ebf2f83a0264e7cb5257abb7942a1ec (127.0.98.3:45521): Not found: Tablet not found: 633f57b3df6c4c5ea74c93ee2c9696b8
I20260618 23:34:19.182899 15288 leader_election.cc:302] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efa7e30dcd9841d993d31ba6b984fd0a; no voters: 0ebf2f83a0264e7cb5257abb7942a1ec, be6c382c0e6f45a5b00483b7291bbd19
I20260618 23:34:19.183063 15671 raft_consensus.cc:2726] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 23:34:19.233074 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:19.232712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.3:48665)
0618 23:34:19.232942 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:19.233020 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":120,"threads_started":1}
I20260618 23:34:20.008500 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:20.008083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.1:51979)
0618 23:34:20.008372 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:20.008444 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":111,"threads_started":1}
I20260618 23:34:20.089056 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:20.088727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.2:37699)
0618 23:34:20.088927 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:20.088997 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":102,"threads_started":1}
I20260618 23:34:20.234076 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:20.233784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.3:60483)
0618 23:34:20.233965 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:20.234027 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":85,"threads_started":1}
I20260618 23:34:21.009553 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:21.009274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.1:49533)
0618 23:34:21.009450 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:21.009503 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":83,"threads_started":1}
I20260618 23:34:21.090019 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:21.089736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.2:35355)
0618 23:34:21.089911 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:21.089965 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":89,"threads_started":1}
I20260618 23:34:21.235076 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:21.234752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.3:34337)
0618 23:34:21.234962 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:21.235028 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":106,"threads_started":1}
I20260618 23:34:21.242118 15682 raft_consensus.cc:482] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:34:21.242269 15682 raft_consensus.cc:504] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } }
I20260618 23:34:21.242535 15682 leader_election.cc:288] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be6c382c0e6f45a5b00483b7291bbd19 (127.0.98.2:39423), 0ebf2f83a0264e7cb5257abb7942a1ec (127.0.98.3:45521)
W20260618 23:34:21.243033 15287 leader_election.cc:341] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be6c382c0e6f45a5b00483b7291bbd19 (127.0.98.2:39423): Not found: Tablet not found: 633f57b3df6c4c5ea74c93ee2c9696b8
W20260618 23:34:21.243120 15288 leader_election.cc:341] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ebf2f83a0264e7cb5257abb7942a1ec (127.0.98.3:45521): Not found: Tablet not found: 633f57b3df6c4c5ea74c93ee2c9696b8
I20260618 23:34:21.243237 15288 leader_election.cc:302] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efa7e30dcd9841d993d31ba6b984fd0a; no voters: 0ebf2f83a0264e7cb5257abb7942a1ec, be6c382c0e6f45a5b00483b7291bbd19
I20260618 23:34:21.243319 15682 raft_consensus.cc:2726] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 23:34:22.010331 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:22.010107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.1:37955)
0618 23:34:22.010247 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:22.010285 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":95,"threads_started":1}
I20260618 23:34:22.090932 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:22.090661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.2:54467)
0618 23:34:22.090816 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:22.090872 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":75,"threads_started":1}
I20260618 23:34:22.235965 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:22.235728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.3:34753)
0618 23:34:22.235880 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:22.235924 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260618 23:34:23.011312 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:23.010978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.1:50323)
0618 23:34:23.011183 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:23.011247 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":85,"threads_started":1}
I20260618 23:34:23.030036 15687 raft_consensus.cc:482] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:34:23.030174 15687 raft_consensus.cc:504] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } }
I20260618 23:34:23.030416 15687 leader_election.cc:288] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be6c382c0e6f45a5b00483b7291bbd19 (127.0.98.2:39423), 0ebf2f83a0264e7cb5257abb7942a1ec (127.0.98.3:45521)
W20260618 23:34:23.030835 15287 leader_election.cc:341] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be6c382c0e6f45a5b00483b7291bbd19 (127.0.98.2:39423): Not found: Tablet not found: 633f57b3df6c4c5ea74c93ee2c9696b8
W20260618 23:34:23.030941 15288 leader_election.cc:341] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ebf2f83a0264e7cb5257abb7942a1ec (127.0.98.3:45521): Not found: Tablet not found: 633f57b3df6c4c5ea74c93ee2c9696b8
I20260618 23:34:23.031051 15288 leader_election.cc:302] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efa7e30dcd9841d993d31ba6b984fd0a; no voters: 0ebf2f83a0264e7cb5257abb7942a1ec, be6c382c0e6f45a5b00483b7291bbd19
I20260618 23:34:23.031134 15687 raft_consensus.cc:2726] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 23:34:23.091988 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:23.091681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.2:51673)
0618 23:34:23.091891 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:23.091942 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":101,"threads_started":1}
I20260618 23:34:23.236829 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:23.236550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.3:42819)
0618 23:34:23.236701 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:23.236757 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":71,"threads_started":1}
I20260618 23:34:24.012203 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:24.011860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.1:46755)
0618 23:34:24.012073 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:24.012136 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":100,"threads_started":1}
I20260618 23:34:24.092931 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:24.092593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.2:37199)
0618 23:34:24.092817 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:24.092889 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":90,"threads_started":1}
I20260618 23:34:24.237692 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:24.237411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.3:39845)
0618 23:34:24.237590 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:24.237639 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":80,"threads_started":1}
I20260618 23:34:25.013164 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:25.012840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.1:56149)
0618 23:34:25.013050 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:25.013111 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":84,"threads_started":1}
I20260618 23:34:25.093801 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:25.093522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.2:38503)
0618 23:34:25.093704 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:25.093753 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":83,"threads_started":1}
I20260618 23:34:25.238737 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:25.238346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.3:36957)
0618 23:34:25.238582 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:25.238662 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260618 23:34:25.611361 15696 raft_consensus.cc:482] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:34:25.611513 15696 raft_consensus.cc:504] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } }
I20260618 23:34:25.611835 15696 leader_election.cc:288] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be6c382c0e6f45a5b00483b7291bbd19 (127.0.98.2:39423), 0ebf2f83a0264e7cb5257abb7942a1ec (127.0.98.3:45521)
W20260618 23:34:25.612417 15288 leader_election.cc:341] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ebf2f83a0264e7cb5257abb7942a1ec (127.0.98.3:45521): Not found: Tablet not found: 633f57b3df6c4c5ea74c93ee2c9696b8
W20260618 23:34:25.612574 15287 leader_election.cc:341] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be6c382c0e6f45a5b00483b7291bbd19 (127.0.98.2:39423): Not found: Tablet not found: 633f57b3df6c4c5ea74c93ee2c9696b8
I20260618 23:34:25.612630 15287 leader_election.cc:302] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efa7e30dcd9841d993d31ba6b984fd0a; no voters: 0ebf2f83a0264e7cb5257abb7942a1ec, be6c382c0e6f45a5b00483b7291bbd19
I20260618 23:34:25.612776 15696 raft_consensus.cc:2726] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 23:34:26.014165 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:26.013817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.1:49925)
0618 23:34:26.014032 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:26.014083 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
I20260618 23:34:26.094619 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:26.094359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.2:37927)
0618 23:34:26.094523 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:26.094569 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":109,"threads_started":1}
I20260618 23:34:26.239661 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:26.239325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.3:57761)
0618 23:34:26.239500 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:26.239601 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":86,"threads_started":1}
I20260618 23:34:27.015275 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:27.014935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.1:58789)
0618 23:34:27.015144 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:27.015210 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":88,"threads_started":1}
I20260618 23:34:27.095443 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:27.095135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.2:33689)
0618 23:34:27.095319 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:27.095379 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":84,"threads_started":1}
I20260618 23:34:27.240693 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:27.240337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.3:47981)
0618 23:34:27.240537 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:27.240631 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":84,"threads_started":1}
I20260618 23:34:28.016294 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:28.015984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.1:59701)
0618 23:34:28.016186 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:28.016241 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":87,"threads_started":1}
I20260618 23:34:28.096418 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:28.096113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.2:48017)
0618 23:34:28.096308 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:28.096362 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":111,"threads_started":1}
I20260618 23:34:28.241734 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:28.241409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.3:42069)
0618 23:34:28.241632 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:28.241683 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":130,"threads_started":1}
I20260618 23:34:29.017330 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:29.016979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.1:35679)
0618 23:34:29.017180 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:29.017251 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":69,"threads_started":1}
I20260618 23:34:29.097363 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:29.097050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.2:56249)
0618 23:34:29.097236 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:29.097299 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
I20260618 23:34:29.242722 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:29.242392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.3:33545)
0618 23:34:29.242597 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:29.242655 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":119,"threads_started":1}
I20260618 23:34:30.018378 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:30.018032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.1:58197)
0618 23:34:30.018240 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:30.018313 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":84,"threads_started":1}
I20260618 23:34:30.098367 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:30.098054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.2:32787)
0618 23:34:30.098246 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:30.098288 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":121,"threads_started":1}
I20260618 23:34:30.243678 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:30.243284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.3:59793)
0618 23:34:30.243482 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:30.243613 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":107,"threads_started":1}
I20260618 23:34:31.019264 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:31.018966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.1:57423)
0618 23:34:31.019162 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:31.019211 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":83,"threads_started":1}
I20260618 23:34:31.099318 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:31.098980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.2:58979)
0618 23:34:31.099182 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:31.099253 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":93,"threads_started":1}
I20260618 23:34:31.244558 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:31.244257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.3:33811)
0618 23:34:31.244463 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:31.244508 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260618 23:34:32.020133 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:32.019865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.1:45907)
0618 23:34:32.020046 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:32.020084 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":92,"threads_started":1}
I20260618 23:34:32.100179 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:32.099931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.2:59149)
0618 23:34:32.100072 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:32.100115 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20260618 23:34:32.245486 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:32.245184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.3:40267)
0618 23:34:32.245361 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:32.245423 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":97,"threads_started":1}
I20260618 23:34:32.730319 15718 raft_consensus.cc:482] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:34:32.730475 15718 raft_consensus.cc:504] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6c382c0e6f45a5b00483b7291bbd19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39423 } } peers { permanent_uuid: "0ebf2f83a0264e7cb5257abb7942a1ec" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45521 } } peers { permanent_uuid: "efa7e30dcd9841d993d31ba6b984fd0a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44499 } }
I20260618 23:34:32.730708 15718 leader_election.cc:288] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers be6c382c0e6f45a5b00483b7291bbd19 (127.0.98.2:39423), 0ebf2f83a0264e7cb5257abb7942a1ec (127.0.98.3:45521)
W20260618 23:34:32.731232 15288 leader_election.cc:341] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0ebf2f83a0264e7cb5257abb7942a1ec (127.0.98.3:45521): Not found: Tablet not found: 633f57b3df6c4c5ea74c93ee2c9696b8
W20260618 23:34:32.731307 15287 leader_election.cc:341] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer be6c382c0e6f45a5b00483b7291bbd19 (127.0.98.2:39423): Not found: Tablet not found: 633f57b3df6c4c5ea74c93ee2c9696b8
I20260618 23:34:32.731420 15287 leader_election.cc:302] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: efa7e30dcd9841d993d31ba6b984fd0a; no voters: 0ebf2f83a0264e7cb5257abb7942a1ec, be6c382c0e6f45a5b00483b7291bbd19
I20260618 23:34:32.731523 15718 raft_consensus.cc:2726] T 633f57b3df6c4c5ea74c93ee2c9696b8 P efa7e30dcd9841d993d31ba6b984fd0a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 23:34:33.021239 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:33.020785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.1:34989)
0618 23:34:33.021138 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:33.021189 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":213,"threads_started":1}
I20260618 23:34:33.101346 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:33.101016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.2:51337)
0618 23:34:33.101213 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:33.101282 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":78,"threads_started":1}
I20260618 23:34:33.240981 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 15240
I20260618 23:34:33.246465 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:33.246202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36203 (local address 127.0.98.3:34993)
0618 23:34:33.246350 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:33.246385 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:36203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":67,"threads_started":1}
I20260618 23:34:33.246557 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 15403
I20260618 23:34:33.252272 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 15536
2026-06-18T23:34:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18404 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:34:33 dist-test-slave-6224 krb5kdc[15724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:34:33 dist-test-slave-6224 krb5kdc[15724](info): set up 2 sockets
Jun 18 23:34:33 dist-test-slave-6224 krb5kdc[15724](info): commencing operation
krb5kdc: starting...
W20260618 23:34:34.343873 392 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:34:34 dist-test-slave-6224 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:34:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:34:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:34:34.474849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38187
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:34:34.566210 15737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:34.566505 15737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:34.566546 15737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:34.567821 15737 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260618 23:34:34.567902 15737 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260618 23:34:34.567924 15737 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260618 23:34:34.567945 15737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:34:34.567972 15737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:34.568005 15737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:34.568029 15737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:34:34.568059 15737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:34:34.569660 15737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:34.569988 15737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:34.570241 15737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:34:34.576124 15737 server_base.cc:1034] running on GCE node
W20260618 23:34:34.576212 15742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:34.576330 15743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:34.576212 15745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:34.576856 15737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:34.577380 15737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:34.578613 15737 hybrid_clock.cc:648] HybridClock initialized: now 1781825674578615 us; error 67 us; skew 500 ppm
Jun 18 23:34:34 dist-test-slave-6224 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:34.581509 15737 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:34:34.582057 15737 webserver.cc:460] Webserver started at http://127.0.98.62:35687/ using document root <none> and password file <none>
I20260618 23:34:34.582333 15737 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:34.582387 15737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:34.582554 15737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:34:34.583802 15737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "85670a759a204c2ab74dae692caffc5b"
format_stamp: "Formatted at 2026-06-18 23:34:34 on dist-test-slave-6224"
server_key: "bc5c96dfa5a9eaa052704edb3d5c2809"
server_key_iv: "3e0e83ad73c466fba07f8a0aabe630a9"
server_key_version: "encryptionkey@0"
I20260618 23:34:34.584192 15737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "85670a759a204c2ab74dae692caffc5b"
format_stamp: "Formatted at 2026-06-18 23:34:34 on dist-test-slave-6224"
server_key: "bc5c96dfa5a9eaa052704edb3d5c2809"
server_key_iv: "3e0e83ad73c466fba07f8a0aabe630a9"
server_key_version: "encryptionkey@0"
I20260618 23:34:34.585862 15737 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:34:34.586876 15752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:34.587188 15737 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:34.587311 15737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "85670a759a204c2ab74dae692caffc5b"
format_stamp: "Formatted at 2026-06-18 23:34:34 on dist-test-slave-6224"
server_key: "bc5c96dfa5a9eaa052704edb3d5c2809"
server_key_iv: "3e0e83ad73c466fba07f8a0aabe630a9"
server_key_version: "encryptionkey@0"
I20260618 23:34:34.587409 15737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:34.619887 15737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:34.622223 15737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:34.622406 15737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:34.627290 15737 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38187
I20260618 23:34:34.627363 15804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38187 every 8 connection(s)
I20260618 23:34:34.627799 15737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:34:34.628849 15805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:34:34.631489 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 15737
I20260618 23:34:34.631657 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:34:34.631825 15805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b: Bootstrap starting.
I20260618 23:34:34.631919 392 external_mini_cluster.cc:1428] Setting key 9676bcf58f83c08a785a64f117760223
I20260618 23:34:34.632699 15805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b: Neither blocks nor log segments found. Creating new log.
I20260618 23:34:34.633104 15805 log.cc:824] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:34.634074 15805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b: No bootstrap required, opened a new log
I20260618 23:34:34.636781 15805 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85670a759a204c2ab74dae692caffc5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38187 } }
I20260618 23:34:34.637004 15805 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:34:34.637043 15805 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85670a759a204c2ab74dae692caffc5b, State: Initialized, Role: FOLLOWER
I20260618 23:34:34.637190 15805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85670a759a204c2ab74dae692caffc5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38187 } }
I20260618 23:34:34.637274 15805 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:34:34.637307 15805 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:34:34.637356 15805 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:34:34.638204 15805 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85670a759a204c2ab74dae692caffc5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38187 } }
I20260618 23:34:34.638340 15805 leader_election.cc:302] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85670a759a204c2ab74dae692caffc5b; no voters:
I20260618 23:34:34.638603 15805 leader_election.cc:288] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:34:34.638645 15810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:34:34.638790 15810 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 1 LEADER]: Becoming Leader. State: Replica: 85670a759a204c2ab74dae692caffc5b, State: Running, Role: LEADER
I20260618 23:34:34.638963 15805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:34:34.638963 15810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85670a759a204c2ab74dae692caffc5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38187 } }
I20260618 23:34:34.639596 15811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85670a759a204c2ab74dae692caffc5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85670a759a204c2ab74dae692caffc5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38187 } } }
I20260618 23:34:34.639756 15811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [sys.catalog]: This master's current role is: LEADER
Jun 18 23:34:34 dist-test-slave-6224 krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:34:34.640637 15812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85670a759a204c2ab74dae692caffc5b. Latest consensus state: current_term: 1 leader_uuid: "85670a759a204c2ab74dae692caffc5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85670a759a204c2ab74dae692caffc5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38187 } } }
I20260618 23:34:34.640846 15819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:34:34.641009 15812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [sys.catalog]: This master's current role is: LEADER
I20260618 23:34:34.641736 15819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:34:34.645354 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:34.632612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43310 (local address 127.0.98.62:38187)
0618 23:34:34.632868 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:34.632883 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:34.632933 (+ 50us) server_negotiation.cc:407] Connection header received
0618 23:34:34.633577 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:34.633585 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:34.633651 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:34.633870 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:34.634582 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:34.636098 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:34.636928 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:34.637582 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:34.637710 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:34.640796 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:34.640803 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:34.640803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:34.640821 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:34.643314 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:34.643828 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:34.643831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:34.643832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:34.643961 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:34.644176 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:34.644176 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:34.644177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:34.644543 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:34.644728 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:34.644876 (+ 148us) server_negotiation.cc:299] Negotiation successful
0618 23:34:34.645068 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":93,"threads_started":1}
I20260618 23:34:34.648101 15819 catalog_manager.cc:1347] Generated new cluster ID: 2b34a157759b4477acbff5f2ae262ed3
I20260618 23:34:34.648212 15819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:34:34.674167 15819 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:34:34.674821 15819 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:34:34.679612 15819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b: Generated new TSK 0
I20260618 23:34:34.679854 15819 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260618 23:34:34.712514 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38187
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:34:34.801168 15831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:34.801403 15831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:34.801429 15831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:34.802666 15831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:34.802726 15831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:34.802755 15831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:34:34.804421 15831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:34.804702 15831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:34.804982 15831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:34.810670 15839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:34.810658 15836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:34.810658 15837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:34.810899 15831 server_base.cc:1034] running on GCE node
I20260618 23:34:34.811324 15831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:34.811707 15831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:34.812899 15831 hybrid_clock.cc:648] HybridClock initialized: now 1781825674812883 us; error 41 us; skew 500 ppm
Jun 18 23:34:34 dist-test-slave-6224 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:34.816031 15831 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:34:34.816577 15831 webserver.cc:460] Webserver started at http://127.0.98.1:38985/ using document root <none> and password file <none>
I20260618 23:34:34.816828 15831 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:34.816880 15831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:34.817023 15831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:34:34.818178 15831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "aa6e88a6a1404de09e8beaf85d5d4deb"
format_stamp: "Formatted at 2026-06-18 23:34:34 on dist-test-slave-6224"
server_key: "b7dacc4ec7279f24a11b02643a1b9f2b"
server_key_iv: "8c3a669a24149b8364408a56a0ec853f"
server_key_version: "encryptionkey@0"
I20260618 23:34:34.818564 15831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "aa6e88a6a1404de09e8beaf85d5d4deb"
format_stamp: "Formatted at 2026-06-18 23:34:34 on dist-test-slave-6224"
server_key: "b7dacc4ec7279f24a11b02643a1b9f2b"
server_key_iv: "8c3a669a24149b8364408a56a0ec853f"
server_key_version: "encryptionkey@0"
I20260618 23:34:34.820091 15831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:34:34.820974 15846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:34.821255 15831 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:34.821367 15831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "aa6e88a6a1404de09e8beaf85d5d4deb"
format_stamp: "Formatted at 2026-06-18 23:34:34 on dist-test-slave-6224"
server_key: "b7dacc4ec7279f24a11b02643a1b9f2b"
server_key_iv: "8c3a669a24149b8364408a56a0ec853f"
server_key_version: "encryptionkey@0"
I20260618 23:34:34.821455 15831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:34.838258 15831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:34.840909 15831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:34.841087 15831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:34.841332 15831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:34:34.841666 15831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:34:34.841723 15831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:34.841763 15831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:34:34.841790 15831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:34.848688 15831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38713
I20260618 23:34:34.848856 15959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38713 every 8 connection(s)
I20260618 23:34:34.849125 15831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
Jun 18 23:34:34 dist-test-slave-6224 krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:34:34.858141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 15831
I20260618 23:34:34.858278 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:34:34.858510 392 external_mini_cluster.cc:1428] Setting key 9df0e664ed0db50e8b31284e1031b501
I20260618 23:34:34.859987 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:34.850088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33167 (local address 127.0.98.62:38187)
0618 23:34:34.850161 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:34.850164 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:34.850444 (+ 280us) server_negotiation.cc:407] Connection header received
0618 23:34:34.850948 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:34.850949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:34.850963 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:34.851034 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:34.852102 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:34.852661 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:34.853680 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:34.853921 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:34.854012 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:34.856525 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:34.856526 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:34.856527 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:34.856536 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:34.857961 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:34.858542 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:34.858543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:34.858544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:34.858636 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:34.859079 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:34.859080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:34.859081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:34.859293 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:34.859358 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:34.859767 (+ 409us) server_negotiation.cc:299] Negotiation successful
0618 23:34:34.859873 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 23:34:34.860267 15962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:34.850062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.1:33167)
0618 23:34:34.850278 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:34.850324 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:34.850843 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:34.851096 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:34.851099 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:34.851782 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:34.851994 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:34.851999 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:34.852757 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:34.852757 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:34.853581 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:34.853583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:34.854025 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:34.854026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:34.854129 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:34.854730 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:34.854739 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:34.856402 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:34.858065 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:34.858073 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:34.858075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:34.858430 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:34.858738 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:34.858740 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:34.858740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:34.859002 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:34.859349 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:34.859357 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:34.859540 (+ 183us) client_negotiation.cc:769] Sending connection context
0618 23:34:34.859732 (+ 192us) client_negotiation.cc:240] Negotiation successful
0618 23:34:34.859960 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":111,"threads_started":1}
I20260618 23:34:34.860873 15960 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38187
I20260618 23:34:34.860986 15960 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:34.861214 15960 heartbeater.cc:511] Master 127.0.98.62:38187 requested a full tablet report, sending...
I20260618 23:34:34.861807 15769 ts_manager.cc:194] Registered new tserver with Master: aa6e88a6a1404de09e8beaf85d5d4deb (127.0.98.1:38713)
I20260618 23:34:34.862498 15769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33167
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260618 23:34:34.884572 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38187
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:34:34.975929 15965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:34.976190 15965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:34.976218 15965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:34.977432 15965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:34.977478 15965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:34.977507 15965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:34:34.979036 15965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:34.979329 15965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:34.979643 15965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:34.985560 15973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:34.985530 15970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:34.985597 15971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:34.985633 15965 server_base.cc:1034] running on GCE node
I20260618 23:34:34.986028 15965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:34.986340 15965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:34.987514 15965 hybrid_clock.cc:648] HybridClock initialized: now 1781825674987487 us; error 56 us; skew 500 ppm
Jun 18 23:34:34 dist-test-slave-6224 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:34.990700 15965 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:34:34.991300 15965 webserver.cc:460] Webserver started at http://127.0.98.2:38369/ using document root <none> and password file <none>
I20260618 23:34:34.991622 15965 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:34.991705 15965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:34.991858 15965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:34:34.993140 15965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "43363f29c3184a7d9f6f916bf560a182"
format_stamp: "Formatted at 2026-06-18 23:34:34 on dist-test-slave-6224"
server_key: "81708018309be01e04845b8534047d5b"
server_key_iv: "0271e2adcae4c10e6cbef54c0b366804"
server_key_version: "encryptionkey@0"
I20260618 23:34:34.993557 15965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "43363f29c3184a7d9f6f916bf560a182"
format_stamp: "Formatted at 2026-06-18 23:34:34 on dist-test-slave-6224"
server_key: "81708018309be01e04845b8534047d5b"
server_key_iv: "0271e2adcae4c10e6cbef54c0b366804"
server_key_version: "encryptionkey@0"
I20260618 23:34:34.995138 15965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:34:34.996245 15980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:34.996549 15965 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260618 23:34:34.996678 15965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "43363f29c3184a7d9f6f916bf560a182"
format_stamp: "Formatted at 2026-06-18 23:34:34 on dist-test-slave-6224"
server_key: "81708018309be01e04845b8534047d5b"
server_key_iv: "0271e2adcae4c10e6cbef54c0b366804"
server_key_version: "encryptionkey@0"
I20260618 23:34:34.996771 15965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:35.017323 15965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:35.020210 15965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:35.020419 15965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:35.020741 15965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:34:35.021162 15965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:34:35.021224 15965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:35.021273 15965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:34:35.021303 15965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:35.029631 15965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44357
I20260618 23:34:35.029846 16093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44357 every 8 connection(s)
I20260618 23:34:35.030157 15965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:34:35.030650 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 15965
I20260618 23:34:35.030781 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:34:35.031052 392 external_mini_cluster.cc:1428] Setting key ab5aaa321ab1ca342eae71af1e2e5771
Jun 18 23:34:35 dist-test-slave-6224 krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260618 23:34:35.043308 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:35.031119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57235 (local address 127.0.98.62:38187)
0618 23:34:35.031176 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:35.031178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:35.031600 (+ 422us) server_negotiation.cc:407] Connection header received
0618 23:34:35.032311 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:35.032312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:35.032327 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:35.032401 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:35.033444 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.034140 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.035751 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.035973 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.036043 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:35.039264 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:35.039266 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:35.039266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:35.039276 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:35.041416 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:35.042058 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:35.042059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:35.042059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:35.042131 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:35.042450 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:35.042450 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:35.042451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:35.042581 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:35.042649 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:35.043123 (+ 474us) server_negotiation.cc:299] Negotiation successful
0618 23:34:35.043224 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260618 23:34:35.043622 16096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:35.031134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.2:57235)
0618 23:34:35.031385 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:35.031441 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:35.032187 (+ 746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:35.032463 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:35.032467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:35.033108 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:35.033337 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:35.033341 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.034248 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.034250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:35.035654 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:35.035658 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.036078 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.036079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:35.036236 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:35.036905 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:35.036916 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:35.039067 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:35.041537 (+ 2470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:35.041550 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:35.041553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:35.041905 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:35.042233 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:35.042234 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:35.042234 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:35.042374 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:35.042679 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:35.042687 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:35.042912 (+ 225us) client_negotiation.cc:769] Sending connection context
0618 23:34:35.043050 (+ 138us) client_negotiation.cc:240] Negotiation successful
0618 23:34:35.043275 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":108,"threads_started":1}
I20260618 23:34:35.044282 16094 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38187
I20260618 23:34:35.044447 16094 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:35.044693 16094 heartbeater.cc:511] Master 127.0.98.62:38187 requested a full tablet report, sending...
I20260618 23:34:35.045148 15769 ts_manager.cc:194] Registered new tserver with Master: 43363f29c3184a7d9f6f916bf560a182 (127.0.98.2:44357)
I20260618 23:34:35.045630 15769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260618 23:34:35.060729 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38187
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:34:35.148216 16099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:35.148445 16099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:35.148470 16099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:35.149561 16099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:35.149605 16099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:35.149631 16099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:34:35.151022 16099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:35.151221 16099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:35.151403 16099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:35.157024 16105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:35.156992 16104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:35.157212 16099 server_base.cc:1034] running on GCE node
W20260618 23:34:35.156980 16107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:35.157661 16099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:35.157979 16099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:35.159166 16099 hybrid_clock.cc:648] HybridClock initialized: now 1781825675159141 us; error 54 us; skew 500 ppm
Jun 18 23:34:35 dist-test-slave-6224 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:35.162379 16099 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:34:35.162945 16099 webserver.cc:460] Webserver started at http://127.0.98.3:36389/ using document root <none> and password file <none>
I20260618 23:34:35.163205 16099 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:35.163267 16099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:35.163470 16099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:34:35.164760 16099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "d41f5d86f61f4ae99111fde71cace9dc"
format_stamp: "Formatted at 2026-06-18 23:34:35 on dist-test-slave-6224"
server_key: "3ca6241ff761b86c0910fcb4e7bff133"
server_key_iv: "7feacaba364bd1d8945d71174d0da67e"
server_key_version: "encryptionkey@0"
I20260618 23:34:35.165266 16099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d41f5d86f61f4ae99111fde71cace9dc"
format_stamp: "Formatted at 2026-06-18 23:34:35 on dist-test-slave-6224"
server_key: "3ca6241ff761b86c0910fcb4e7bff133"
server_key_iv: "7feacaba364bd1d8945d71174d0da67e"
server_key_version: "encryptionkey@0"
I20260618 23:34:35.166942 16099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:34:35.167937 16114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:35.168191 16099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:35.168305 16099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "d41f5d86f61f4ae99111fde71cace9dc"
format_stamp: "Formatted at 2026-06-18 23:34:35 on dist-test-slave-6224"
server_key: "3ca6241ff761b86c0910fcb4e7bff133"
server_key_iv: "7feacaba364bd1d8945d71174d0da67e"
server_key_version: "encryptionkey@0"
I20260618 23:34:35.168385 16099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:35.178622 16099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:35.181093 16099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:35.181250 16099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:35.181560 16099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:34:35.181910 16099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:34:35.181967 16099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:35.182011 16099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:34:35.182039 16099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:35.189191 16099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44033
I20260618 23:34:35.189248 16227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44033 every 8 connection(s)
I20260618 23:34:35.189577 16099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
Jun 18 23:34:35 dist-test-slave-6224 krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:34:35.197191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 16099
I20260618 23:34:35.197314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:34:35.197541 392 external_mini_cluster.cc:1428] Setting key 168c0e35dd4b9246233ad69ecd95db19
I20260618 23:34:35.201272 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:35.190464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34833 (local address 127.0.98.62:38187)
0618 23:34:35.190546 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:35.190548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:35.190908 (+ 360us) server_negotiation.cc:407] Connection header received
0618 23:34:35.191467 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:35.191468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:35.191481 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:35.191599 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:35.192829 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.193395 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.194561 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.194757 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.194826 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:35.197649 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:35.197650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:35.197651 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:35.197658 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:35.199633 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:35.200168 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:35.200169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:35.200169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:35.200251 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:35.200511 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:35.200512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:35.200512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:35.200658 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:35.200720 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:35.201095 (+ 375us) server_negotiation.cc:299] Negotiation successful
0618 23:34:35.201184 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 23:34:35.201330 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:35.190480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.3:34833)
0618 23:34:35.190699 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:35.190740 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:35.191357 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:35.191749 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:35.191754 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:35.192453 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:35.192701 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:35.192707 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.193508 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.193509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:35.194468 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:35.194471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.194846 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.194847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:35.194963 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:35.195680 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:35.195690 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:35.197518 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:35.199729 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:35.199736 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:35.199738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:35.200037 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:35.200331 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:35.200332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:35.200332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:35.200439 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:35.200726 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:35.200731 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:35.200901 (+ 170us) client_negotiation.cc:769] Sending connection context
0618 23:34:35.201000 (+ 99us) client_negotiation.cc:240] Negotiation successful
0618 23:34:35.201145 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":117,"threads_started":1}
I20260618 23:34:35.201999 16228 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38187
I20260618 23:34:35.202088 16228 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:35.202324 16228 heartbeater.cc:511] Master 127.0.98.62:38187 requested a full tablet report, sending...
I20260618 23:34:35.202839 15769 ts_manager.cc:194] Registered new tserver with Master: d41f5d86f61f4ae99111fde71cace9dc (127.0.98.3:44033)
I20260618 23:34:35.203405 15769 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34833
I20260618 23:34:35.209985 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:34:35.217967 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:35.211413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43316 (local address 127.0.98.62:38187)
0618 23:34:35.211504 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:35.211506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:35.211603 (+ 97us) server_negotiation.cc:407] Connection header received
0618 23:34:35.211694 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:35.211695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:35.211706 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:35.211792 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:35.212340 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.212875 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.213607 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.213795 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.213851 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:35.215107 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:35.215108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:35.215109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:35.215125 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:35.216765 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:35.217147 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:35.217148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:35.217148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:35.217221 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:35.217420 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:35.217421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:35.217421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:35.217600 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:35.217706 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:35.217807 (+ 101us) server_negotiation.cc:299] Negotiation successful
0618 23:34:35.217884 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 23:34:35.220036 15769 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43316:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:34:35.221999 15769 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:34:35.229640 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:35.225561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36150 (local address 127.0.98.2:44357)
0618 23:34:35.225762 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:35.225767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:35.225777 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:34:35.225818 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:35.225820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:35.225835 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:35.225913 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:34:35.226194 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.227306 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.228619 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.229174 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.229269 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:35.229340 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:35.229490 (+ 150us) server_negotiation.cc:299] Negotiation successful
0618 23:34:35.229540 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":101,"threads_started":1}
I20260618 23:34:35.229627 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:35.225006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44357 (local address 127.0.0.1:36150)
0618 23:34:35.225191 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:35.225244 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:35.225318 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:35.226030 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:35.226031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:35.226034 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:34:35.226113 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:35.226116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.227436 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.227437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:35.228525 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:35.228527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.229268 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.229269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:35.229369 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:35.229386 (+ 17us) client_negotiation.cc:769] Sending connection context
0618 23:34:35.229464 (+ 78us) client_negotiation.cc:240] Negotiation successful
0618 23:34:35.229538 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"mutex_wait_us":1,"thread_start_us":70,"threads_started":1}
I20260618 23:34:35.229947 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:35.225319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38713 (local address 127.0.0.1:58988)
0618 23:34:35.226079 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:35.226091 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:35.226173 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:35.226384 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:35.226385 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:35.226387 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:34:35.226433 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:35.226435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.227690 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.227691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:35.228712 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:35.228714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.229663 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.229664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:35.229718 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:35.229722 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:34:35.229770 (+ 48us) client_negotiation.cc:240] Negotiation successful
0618 23:34:35.229809 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
I20260618 23:34:35.230047 16240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:35.225067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44033 (local address 127.0.0.1:53086)
0618 23:34:35.225247 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:35.225260 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:35.225322 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:35.225598 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:35.225603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:35.225607 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:34:35.225715 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:35.225718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.227016 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.227017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:35.228322 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:35.228325 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.229409 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.229410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:35.229486 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:35.229491 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:34:35.229549 (+ 58us) client_negotiation.cc:240] Negotiation successful
0618 23:34:35.229620 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"mutex_wait_us":1,"thread_start_us":129,"threads_started":1}
I20260618 23:34:35.229821 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:35.225182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53086 (local address 127.0.98.3:44033)
0618 23:34:35.225404 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:35.225409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:35.225420 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:34:35.225469 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:35.225471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:35.225482 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:35.225585 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:34:35.225840 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.226899 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.228438 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.229214 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.229291 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:35.229351 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:35.229672 (+ 321us) server_negotiation.cc:299] Negotiation successful
0618 23:34:35.229721 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":97,"threads_started":1}
I20260618 23:34:35.230238 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:35.225877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58988 (local address 127.0.98.1:38713)
0618 23:34:35.226113 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:35.226118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:35.226137 (+ 19us) server_negotiation.cc:407] Connection header received
0618 23:34:35.226257 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:35.226260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:35.226275 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:35.226365 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:34:35.226500 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.227576 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.228816 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.229525 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.229583 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:35.229630 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:35.230095 (+ 465us) server_negotiation.cc:299] Negotiation successful
0618 23:34:35.230143 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":104,"threads_started":1}
I20260618 23:34:35.231021 16162 tablet_service.cc:1467] Processing CreateTablet for tablet 95777b903b8b4b91a2108e4b547dd393 (DEFAULT_TABLE table=test-table [id=6939179ec1cb4f84a5724a91fc15d4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:34:35.231020 15894 tablet_service.cc:1467] Processing CreateTablet for tablet 95777b903b8b4b91a2108e4b547dd393 (DEFAULT_TABLE table=test-table [id=6939179ec1cb4f84a5724a91fc15d4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:34:35.231412 16162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95777b903b8b4b91a2108e4b547dd393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:34:35.231421 15894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95777b903b8b4b91a2108e4b547dd393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:34:35.231443 16028 tablet_service.cc:1467] Processing CreateTablet for tablet 95777b903b8b4b91a2108e4b547dd393 (DEFAULT_TABLE table=test-table [id=6939179ec1cb4f84a5724a91fc15d4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:34:35.231904 16028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95777b903b8b4b91a2108e4b547dd393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:34:35.235188 16246 tablet_bootstrap.cc:492] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb: Bootstrap starting.
I20260618 23:34:35.235198 16245 tablet_bootstrap.cc:492] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182: Bootstrap starting.
I20260618 23:34:35.235888 16246 tablet_bootstrap.cc:654] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb: Neither blocks nor log segments found. Creating new log.
I20260618 23:34:35.235991 16245 tablet_bootstrap.cc:654] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182: Neither blocks nor log segments found. Creating new log.
I20260618 23:34:35.236423 16246 log.cc:824] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:35.236572 16245 log.cc:824] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:35.236778 16247 tablet_bootstrap.cc:492] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc: Bootstrap starting.
I20260618 23:34:35.237160 16246 tablet_bootstrap.cc:492] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb: No bootstrap required, opened a new log
I20260618 23:34:35.237218 16246 ts_tablet_manager.cc:1397] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260618 23:34:35.237389 16245 tablet_bootstrap.cc:492] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182: No bootstrap required, opened a new log
I20260618 23:34:35.237473 16245 ts_tablet_manager.cc:1397] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260618 23:34:35.238750 16246 raft_consensus.cc:348] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:34:35.238973 16246 raft_consensus.cc:374] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:34:35.239300 16245 raft_consensus.cc:348] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:34:35.239455 16245 raft_consensus.cc:374] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:34:35.239480 16245 raft_consensus.cc:729] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43363f29c3184a7d9f6f916bf560a182, State: Initialized, Role: FOLLOWER
I20260618 23:34:35.239648 16247 tablet_bootstrap.cc:654] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc: Neither blocks nor log segments found. Creating new log.
I20260618 23:34:35.239645 16245 consensus_queue.cc:260] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:34:35.240007 16245 ts_tablet_manager.cc:1428] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260618 23:34:35.240155 16094 heartbeater.cc:503] Master 127.0.98.62:38187 was elected leader, sending a full tablet report...
I20260618 23:34:35.240469 16246 raft_consensus.cc:729] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6e88a6a1404de09e8beaf85d5d4deb, State: Initialized, Role: FOLLOWER
I20260618 23:34:35.240671 16246 consensus_queue.cc:260] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:34:35.241009 16246 ts_tablet_manager.cc:1428] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260618 23:34:35.241093 15960 heartbeater.cc:503] Master 127.0.98.62:38187 was elected leader, sending a full tablet report...
I20260618 23:34:35.241389 16247 log.cc:824] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:35.242225 16247 tablet_bootstrap.cc:492] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc: No bootstrap required, opened a new log
I20260618 23:34:35.242322 16247 ts_tablet_manager.cc:1397] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20260618 23:34:35.244249 16247 raft_consensus.cc:348] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:34:35.244419 16247 raft_consensus.cc:374] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:34:35.244443 16247 raft_consensus.cc:729] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d41f5d86f61f4ae99111fde71cace9dc, State: Initialized, Role: FOLLOWER
I20260618 23:34:35.244593 16247 consensus_queue.cc:260] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:34:35.244920 16247 ts_tablet_manager.cc:1428] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260618 23:34:35.244995 16228 heartbeater.cc:503] Master 127.0.98.62:38187 was elected leader, sending a full tablet report...
W20260618 23:34:35.280856 16095 tablet.cc:2369] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:34:35.282717 16253 raft_consensus.cc:482] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:34:35.282840 16253 raft_consensus.cc:504] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:34:35.283216 16253 leader_election.cc:288] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43363f29c3184a7d9f6f916bf560a182 (127.0.98.2:44357), aa6e88a6a1404de09e8beaf85d5d4deb (127.0.98.1:38713)
I20260618 23:34:35.286859 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:35.283542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47755 (local address 127.0.98.1:38713)
0618 23:34:35.283669 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:35.283672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:35.283682 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:34:35.283712 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:35.283713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:35.283727 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:35.283860 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:34:35.284104 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.284687 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.285868 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.286439 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.286518 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:35.286531 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:35.286735 (+ 204us) server_negotiation.cc:299] Negotiation successful
0618 23:34:35.286782 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260618 23:34:35.286896 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:35.283437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38713 (local address 127.0.98.3:47755)
0618 23:34:35.283510 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:35.283523 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:35.283605 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:35.283883 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:35.283884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:35.283887 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:34:35.283994 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:35.283997 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.284787 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.284789 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:35.285741 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:35.285744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.286543 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.286544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:35.286628 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:35.286633 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:34:35.286699 (+ 66us) client_negotiation.cc:240] Negotiation successful
0618 23:34:35.286784 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":59}
I20260618 23:34:35.287051 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:35.283470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44357 (local address 127.0.98.3:44085)
0618 23:34:35.283763 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:35.283774 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:35.283835 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:35.284076 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:35.284078 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:35.284081 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:34:35.284171 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:35.284174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.284980 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.284981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:35.285907 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:35.285910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.286782 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.286783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:35.286854 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:35.286858 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:34:35.286920 (+ 62us) client_negotiation.cc:240] Negotiation successful
0618 23:34:35.286970 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":10,"thread_start_us":210,"threads_started":1}
I20260618 23:34:35.287074 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:35.283690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44085 (local address 127.0.98.2:44357)
0618 23:34:35.283751 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:35.283754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:35.283845 (+ 91us) server_negotiation.cc:407] Connection header received
0618 23:34:35.283889 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:35.283890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:35.283905 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:35.283992 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:34:35.284298 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.284878 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.286042 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.286659 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.286759 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:35.286770 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:35.286947 (+ 177us) server_negotiation.cc:299] Negotiation successful
0618 23:34:35.286997 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260618 23:34:35.287428 15914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95777b903b8b4b91a2108e4b547dd393" candidate_uuid: "d41f5d86f61f4ae99111fde71cace9dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" is_pre_election: true
I20260618 23:34:35.287513 16048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95777b903b8b4b91a2108e4b547dd393" candidate_uuid: "d41f5d86f61f4ae99111fde71cace9dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43363f29c3184a7d9f6f916bf560a182" is_pre_election: true
I20260618 23:34:35.287684 15914 raft_consensus.cc:2454] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d41f5d86f61f4ae99111fde71cace9dc in term 0.
I20260618 23:34:35.287756 16048 raft_consensus.cc:2454] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d41f5d86f61f4ae99111fde71cace9dc in term 0.
I20260618 23:34:35.287974 16118 leader_election.cc:302] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43363f29c3184a7d9f6f916bf560a182, d41f5d86f61f4ae99111fde71cace9dc; no voters:
I20260618 23:34:35.288184 16253 raft_consensus.cc:2781] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:34:35.288259 16253 raft_consensus.cc:482] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:34:35.288285 16253 raft_consensus.cc:3037] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:34:35.289194 16253 raft_consensus.cc:504] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:34:35.289381 16253 leader_election.cc:288] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc [CANDIDATE]: Term 1 election: Requested vote from peers 43363f29c3184a7d9f6f916bf560a182 (127.0.98.2:44357), aa6e88a6a1404de09e8beaf85d5d4deb (127.0.98.1:38713)
I20260618 23:34:35.289505 16048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95777b903b8b4b91a2108e4b547dd393" candidate_uuid: "d41f5d86f61f4ae99111fde71cace9dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43363f29c3184a7d9f6f916bf560a182"
I20260618 23:34:35.289616 16048 raft_consensus.cc:3037] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:34:35.289801 15914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95777b903b8b4b91a2108e4b547dd393" candidate_uuid: "d41f5d86f61f4ae99111fde71cace9dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb"
I20260618 23:34:35.289947 15914 raft_consensus.cc:3037] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:34:35.290753 16048 raft_consensus.cc:2454] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d41f5d86f61f4ae99111fde71cace9dc in term 1.
I20260618 23:34:35.290948 16118 leader_election.cc:302] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43363f29c3184a7d9f6f916bf560a182, d41f5d86f61f4ae99111fde71cace9dc; no voters:
I20260618 23:34:35.290940 15914 raft_consensus.cc:2454] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d41f5d86f61f4ae99111fde71cace9dc in term 1.
I20260618 23:34:35.291054 16253 raft_consensus.cc:2781] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:34:35.291288 16253 raft_consensus.cc:686] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc [term 1 LEADER]: Becoming Leader. State: Replica: d41f5d86f61f4ae99111fde71cace9dc, State: Running, Role: LEADER
I20260618 23:34:35.291425 16253 consensus_queue.cc:237] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:34:35.292451 15768 catalog_manager.cc:5495] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc reported cstate change: term changed from 0 to 1, leader changed from <none> to d41f5d86f61f4ae99111fde71cace9dc (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d41f5d86f61f4ae99111fde71cace9dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } health_report { overall_health: UNKNOWN } } }
I20260618 23:34:35.307235 16241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:35.304445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53096 (local address 127.0.98.3:44033)
0618 23:34:35.304549 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:35.304551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:35.304562 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:34:35.304612 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:35.304614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:35.304640 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:35.304722 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:34:35.304865 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.305410 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.306211 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:35.306430 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:35.306494 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:35.306705 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:34:35.306839 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:34:35.306926 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:35.306968 (+ 42us) server_negotiation.cc:299] Negotiation successful
0618 23:34:35.307005 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260618 23:34:35.309564 16048 raft_consensus.cc:1261] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182 [term 1 FOLLOWER]: Refusing update from remote peer d41f5d86f61f4ae99111fde71cace9dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:34:35.309638 15914 raft_consensus.cc:1261] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Refusing update from remote peer d41f5d86f61f4ae99111fde71cace9dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:34:35.310048 16253 consensus_queue.cc:1035] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:34:35.310384 16253 consensus_queue.cc:1035] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:34:35.313702 16259 mvcc.cc:205] Tried to move back new op lower bound from 7298357966064668672 to 7298357965994528768. Current Snapshot: MvccSnapshot[applied={T|T < 7298357966064668672 or (T in {7298357966064668672})}]
I20260618 23:34:35.317068 15768 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:34:35.317167 15768 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43316:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:34:35.318151 15768 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b: Sending DeleteTablet for 3 replicas of tablet 95777b903b8b4b91a2108e4b547dd393
I20260618 23:34:35.318522 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 15831
I20260618 23:34:35.318511 16162 tablet_service.cc:1514] Processing DeleteTablet for tablet 95777b903b8b4b91a2108e4b547dd393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:34:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53086
I20260618 23:34:35.318528 16028 tablet_service.cc:1514] Processing DeleteTablet for tablet 95777b903b8b4b91a2108e4b547dd393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:34:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36150
I20260618 23:34:35.319396 16268 tablet_replica.cc:330] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc: stopping tablet replica
I20260618 23:34:35.319511 16268 raft_consensus.cc:2229] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:34:35.319701 16268 raft_consensus.cc:2258] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:34:35.320180 16268 ts_tablet_manager.cc:1905] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:34:35.321741 16268 ts_tablet_manager.cc:1918] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:34:35.321841 16268 log.cc:1196] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/95777b903b8b4b91a2108e4b547dd393
I20260618 23:34:35.322100 16268 ts_tablet_manager.cc:1939] T 95777b903b8b4b91a2108e4b547dd393 P d41f5d86f61f4ae99111fde71cace9dc: Deleting consensus metadata
I20260618 23:34:35.322590 15755 catalog_manager.cc:4841] TS d41f5d86f61f4ae99111fde71cace9dc (127.0.98.3:44033): tablet 95777b903b8b4b91a2108e4b547dd393 (table test-table [id=6939179ec1cb4f84a5724a91fc15d4e8]) successfully deleted
W20260618 23:34:35.325313 15755 connection.cc:535] client connection to 127.0.98.1:38713 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260618 23:34:35.325425 15755 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260618 23:34:35.325495 16269 tablet_replica.cc:330] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182: stopping tablet replica
I20260618 23:34:35.325630 16269 raft_consensus.cc:2229] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:34:35.325762 16269 raft_consensus.cc:2258] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:34:35.325886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 15965
I20260618 23:34:35.326793 16240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:35.326119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38713 (local address 127.0.0.1:59002)
0618 23:34:35.326700 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:35.326746 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":552}
W20260618 23:34:35.328286 15755 catalog_manager.cc:4584] TS aa6e88a6a1404de09e8beaf85d5d4deb (127.0.98.1:38713): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 95777b903b8b4b91a2108e4b547dd393: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38713: connect: Connection refused (error 111)
I20260618 23:34:35.332945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 16099
I20260618 23:34:35.333086 16240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:35.332868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44357 (local address 127.0.0.1:36166)
0618 23:34:35.332972 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:35.333034 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260618 23:34:35.339228 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 15737
I20260618 23:34:55.346103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38187
--webserver_interface=127.0.98.62
--webserver_port=35687
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:34:55.427886 16271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:55.428126 16271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:55.428150 16271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:55.429299 16271 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260618 23:34:55.429343 16271 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260618 23:34:55.429358 16271 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260618 23:34:55.429368 16271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:34:55.429381 16271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:55.429395 16271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:55.429407 16271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:34:55.429419 16271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:34:55.430900 16271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:55.431149 16271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:55.431329 16271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:34:55.441394 16271 server_base.cc:1034] running on GCE node
W20260618 23:34:55.441394 16277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:55.441447 16278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:55.441423 16280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:55.442066 16271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:55.442445 16271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:55.443626 16271 hybrid_clock.cc:648] HybridClock initialized: now 1781825695443610 us; error 59 us; skew 500 ppm
Jun 18 23:34:55 dist-test-slave-6224 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:55.450742 16271 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:34:55.451365 16271 webserver.cc:460] Webserver started at http://127.0.98.62:35687/ using document root <none> and password file <none>
I20260618 23:34:55.451761 16271 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:55.451867 16271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:55.453908 16271 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260618 23:34:55.456290 16287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:55.456640 16271 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260618 23:34:55.456774 16271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "85670a759a204c2ab74dae692caffc5b"
format_stamp: "Formatted at 2026-06-18 23:34:34 on dist-test-slave-6224"
server_key: "bc5c96dfa5a9eaa052704edb3d5c2809"
server_key_iv: "3e0e83ad73c466fba07f8a0aabe630a9"
server_key_version: "encryptionkey@0"
I20260618 23:34:55.457235 16271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:55.479151 16271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:55.482250 16271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:55.482393 16271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:55.487502 16271 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38187
I20260618 23:34:55.487638 16339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38187 every 8 connection(s)
I20260618 23:34:55.488101 16271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:34:55.490036 16340 sys_catalog.cc:263] Verifying existing consensus state
I20260618 23:34:55.490805 16340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b: Bootstrap starting.
I20260618 23:34:55.492929 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 16271
I20260618 23:34:55.493386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38713
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38187
--webserver_port=38985
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:34:55.502737 16340 log.cc:824] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:55.505206 16340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:34:55.505581 16340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b: Bootstrap complete.
I20260618 23:34:55.508495 16340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85670a759a204c2ab74dae692caffc5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38187 } }
I20260618 23:34:55.508759 16340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85670a759a204c2ab74dae692caffc5b, State: Initialized, Role: FOLLOWER
I20260618 23:34:55.508952 16340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85670a759a204c2ab74dae692caffc5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38187 } }
I20260618 23:34:55.509058 16340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:34:55.509123 16340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:34:55.509178 16340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 1 FOLLOWER]: Advancing to term 2
I20260618 23:34:55.510527 16340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85670a759a204c2ab74dae692caffc5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38187 } }
I20260618 23:34:55.510668 16340 leader_election.cc:302] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85670a759a204c2ab74dae692caffc5b; no voters:
I20260618 23:34:55.510926 16340 leader_election.cc:288] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260618 23:34:55.510969 16344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 2 FOLLOWER]: Leader election won for term 2
I20260618 23:34:55.511113 16344 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 2 LEADER]: Becoming Leader. State: Replica: 85670a759a204c2ab74dae692caffc5b, State: Running, Role: LEADER
I20260618 23:34:55.511265 16340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:34:55.511380 16344 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85670a759a204c2ab74dae692caffc5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38187 } }
I20260618 23:34:55.512197 16345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "85670a759a204c2ab74dae692caffc5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85670a759a204c2ab74dae692caffc5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38187 } } }
I20260618 23:34:55.512332 16345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [sys.catalog]: This master's current role is: LEADER
I20260618 23:34:55.512115 16346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85670a759a204c2ab74dae692caffc5b. Latest consensus state: current_term: 2 leader_uuid: "85670a759a204c2ab74dae692caffc5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85670a759a204c2ab74dae692caffc5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38187 } } }
I20260618 23:34:55.512591 16346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [sys.catalog]: This master's current role is: LEADER
I20260618 23:34:55.513185 16354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:34:55.514174 16354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:34:55.514302 16354 catalog_manager.cc:1259] Loaded cluster ID: 2b34a157759b4477acbff5f2ae262ed3
I20260618 23:34:55.514546 16354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:34:55.516038 16354 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:34:55.516281 16354 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b: Loaded TSK: 0
I20260618 23:34:55.531349 16354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b: Generated new TSK 1
I20260618 23:34:55.531469 16354 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 23:34:55.582888 16342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:55.583135 16342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:55.583173 16342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:55.584329 16342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:55.584401 16342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:55.584448 16342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:34:55.585906 16342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38985
--tserver_master_addrs=127.0.98.62:38187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:55.586167 16342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:55.586400 16342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:34:55.591708 16342 server_base.cc:1034] running on GCE node
W20260618 23:34:55.591681 16367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:55.591816 16368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:55.591678 16370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:55.592303 16342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:55.592571 16342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:55.593763 16342 hybrid_clock.cc:648] HybridClock initialized: now 1781825695593731 us; error 53 us; skew 500 ppm
Jun 18 23:34:55 dist-test-slave-6224 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:55.596691 16342 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:34:55.597262 16342 webserver.cc:460] Webserver started at http://127.0.98.1:38985/ using document root <none> and password file <none>
I20260618 23:34:55.597524 16342 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:55.597589 16342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:55.599041 16342 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:34:55.599787 16377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:55.600036 16342 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:55.600194 16342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "aa6e88a6a1404de09e8beaf85d5d4deb"
format_stamp: "Formatted at 2026-06-18 23:34:34 on dist-test-slave-6224"
server_key: "b7dacc4ec7279f24a11b02643a1b9f2b"
server_key_iv: "8c3a669a24149b8364408a56a0ec853f"
server_key_version: "encryptionkey@0"
I20260618 23:34:55.600800 16342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:55.618245 16342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:55.621333 16342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:55.621513 16342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:55.621860 16342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:34:55.622512 16384 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260618 23:34:55.624239 16342 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 23:34:55.624321 16342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260618 23:34:55.624362 16342 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 23:34:55.625470 16342 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 23:34:55.625525 16342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260618 23:34:55.625639 16384 tablet_bootstrap.cc:492] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb: Bootstrap starting.
I20260618 23:34:55.634552 16342 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38713
I20260618 23:34:55.635188 16342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:34:55.636106 16491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38713 every 8 connection(s)
I20260618 23:34:55.637229 16384 log.cc:824] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:55.640033 16384 tablet_bootstrap.cc:492] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:34:55.640291 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 16342
I20260618 23:34:55.640635 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44357
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38187
--webserver_port=38369
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:34:55.640916 16384 tablet_bootstrap.cc:492] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb: Bootstrap complete.
I20260618 23:34:55.642130 16384 ts_tablet_manager.cc:1397] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.010s
I20260618 23:34:55.643496 16384 raft_consensus.cc:348] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:34:55.643886 16384 raft_consensus.cc:729] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6e88a6a1404de09e8beaf85d5d4deb, State: Initialized, Role: FOLLOWER
I20260618 23:34:55.644107 16384 consensus_queue.cc:260] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:34:55.644412 16384 ts_tablet_manager.cc:1428] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
Jun 18 23:34:55 dist-test-slave-6224 krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:34:55.650889 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:55.636244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37787 (local address 127.0.98.62:38187)
0618 23:34:55.637623 (+ 1379us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:55.637631 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:55.637664 (+ 33us) server_negotiation.cc:407] Connection header received
0618 23:34:55.637788 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:55.637793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:55.637832 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:55.638005 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:55.639016 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.640524 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.642232 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.642825 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.642959 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:55.645982 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:55.645987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:55.645987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:55.646004 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:55.648473 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:55.649094 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:55.649098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:55.649098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:55.649239 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:55.649593 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:55.649594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:55.649594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:55.649952 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:55.650053 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:55.650488 (+ 435us) server_negotiation.cc:299] Negotiation successful
0618 23:34:55.650648 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1319,"thread_start_us":91,"threads_started":1}
I20260618 23:34:55.650904 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:55.636158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.1:37787)
0618 23:34:55.636451 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:55.636506 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:55.637350 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:55.638083 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:55.638089 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:55.638529 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:55.638834 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:55.638842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.640681 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.640682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:55.642082 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:55.642086 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.642970 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.642972 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:55.643143 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:55.643890 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:55.643900 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:55.645822 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:55.648595 (+ 2773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:55.648604 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:55.648607 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:55.648969 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:55.649336 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:55.649337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:55.649338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:55.649497 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:55.650049 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:55.650056 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:55.650261 (+ 205us) client_negotiation.cc:769] Sending connection context
0618 23:34:55.650405 (+ 144us) client_negotiation.cc:240] Negotiation successful
0618 23:34:55.650618 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":125,"threads_started":1}
I20260618 23:34:55.652467 16492 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38187
I20260618 23:34:55.652590 16492 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:55.652813 16492 heartbeater.cc:511] Master 127.0.98.62:38187 requested a full tablet report, sending...
I20260618 23:34:55.653820 16304 ts_manager.cc:194] Registered new tserver with Master: aa6e88a6a1404de09e8beaf85d5d4deb (127.0.98.1:38713)
I20260618 23:34:55.655115 16304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37787
I20260618 23:34:55.656322 16492 heartbeater.cc:503] Master 127.0.98.62:38187 was elected leader, sending a full tablet report...
Jun 18 23:34:55 dist-test-slave-6224 krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260618 23:34:55.666863 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:55.654596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38713 (local address 127.0.0.1:33386)
0618 23:34:55.655110 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:55.655148 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:55.655219 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:55.655527 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:55.655532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:55.656774 (+ 1242us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:55.656872 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:55.656875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.658207 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.658208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:55.659730 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:55.659734 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.660432 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.660437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:55.660591 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:55.661288 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:55.661296 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:55.663383 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:55.665555 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:55.665563 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:55.665566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:55.665835 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:55.666074 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:55.666075 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:55.666075 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:55.666142 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:55.666470 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:55.666476 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:55.666536 (+ 60us) client_negotiation.cc:769] Sending connection context
0618 23:34:55.666613 (+ 77us) client_negotiation.cc:240] Negotiation successful
0618 23:34:55.666762 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":117,"threads_started":1}
I20260618 23:34:55.667321 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:55.654887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33386 (local address 127.0.98.1:38713)
0618 23:34:55.655341 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:55.655345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:55.655354 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:34:55.655386 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:55.655388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:55.655401 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:55.655460 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:55.656978 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.658090 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.659841 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.660338 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.660421 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:55.663499 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:55.663506 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:55.663507 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:55.663515 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:55.665455 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:55.665935 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:55.665938 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:55.665939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:55.665996 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:55.666203 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:55.666204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:55.666204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:55.666373 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:55.666444 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:55.667129 (+ 685us) server_negotiation.cc:299] Negotiation successful
0618 23:34:55.667228 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":122,"threads_started":1}
I20260618 23:34:55.667649 16426 tablet_service.cc:1514] Processing DeleteTablet for tablet 95777b903b8b4b91a2108e4b547dd393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:34:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33386
I20260618 23:34:55.668291 16501 tablet_replica.cc:330] stopping tablet replica
I20260618 23:34:55.668397 16501 raft_consensus.cc:2229] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:34:55.668469 16501 raft_consensus.cc:2258] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:34:55.668834 16501 ts_tablet_manager.cc:1905] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:34:55.671151 16501 ts_tablet_manager.cc:1918] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:34:55.671227 16501 log.cc:1196] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/wal/wals/95777b903b8b4b91a2108e4b547dd393
I20260618 23:34:55.671443 16501 ts_tablet_manager.cc:1939] T 95777b903b8b4b91a2108e4b547dd393 P aa6e88a6a1404de09e8beaf85d5d4deb: Deleting consensus metadata
I20260618 23:34:55.672067 16290 catalog_manager.cc:4841] TS aa6e88a6a1404de09e8beaf85d5d4deb (127.0.98.1:38713): tablet 95777b903b8b4b91a2108e4b547dd393 (table test-table [id=6939179ec1cb4f84a5724a91fc15d4e8]) successfully deleted
W20260618 23:34:55.735056 16497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:55.735278 16497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:55.735303 16497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:55.736413 16497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:55.736461 16497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:55.736487 16497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:34:55.737885 16497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38369
--tserver_master_addrs=127.0.98.62:38187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:55.738080 16497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:55.738265 16497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:55.743780 16506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:55.743842 16507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:55.744053 16497 server_base.cc:1034] running on GCE node
W20260618 23:34:55.743785 16509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:55.744402 16497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:55.744732 16497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:55.745889 16497 hybrid_clock.cc:648] HybridClock initialized: now 1781825695745866 us; error 50 us; skew 500 ppm
Jun 18 23:34:55 dist-test-slave-6224 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:55.748710 16497 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:34:55.749220 16497 webserver.cc:460] Webserver started at http://127.0.98.2:38369/ using document root <none> and password file <none>
I20260618 23:34:55.749461 16497 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:55.749547 16497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:55.751026 16497 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:34:55.751950 16516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:55.752204 16497 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:34:55.752321 16497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "43363f29c3184a7d9f6f916bf560a182"
format_stamp: "Formatted at 2026-06-18 23:34:34 on dist-test-slave-6224"
server_key: "81708018309be01e04845b8534047d5b"
server_key_iv: "0271e2adcae4c10e6cbef54c0b366804"
server_key_version: "encryptionkey@0"
I20260618 23:34:55.752702 16497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:55.762977 16497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:55.765337 16497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:55.765511 16497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:55.765748 16497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:34:55.766285 16523 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260618 23:34:55.768146 16497 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 23:34:55.768226 16497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260618 23:34:55.768273 16497 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 23:34:55.769065 16497 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 23:34:55.769124 16497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260618 23:34:55.769340 16523 tablet_bootstrap.cc:492] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182: Bootstrap starting.
I20260618 23:34:55.776880 16497 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44357
I20260618 23:34:55.777067 16630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44357 every 8 connection(s)
I20260618 23:34:55.777355 16497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:34:55.779623 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 16497
I20260618 23:34:55.779935 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44033
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38187
--webserver_port=36389
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:34:55.781869 16523 log.cc:824] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182: Log is configured to *not* fsync() on all Append() calls
Jun 18 23:34:55 dist-test-slave-6224 krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:34:55.786417 16523 tablet_bootstrap.cc:492] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:34:55.786865 16523 tablet_bootstrap.cc:492] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182: Bootstrap complete.
I20260618 23:34:55.788250 16523 ts_tablet_manager.cc:1397] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.008s
I20260618 23:34:55.789566 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:55.778195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50955 (local address 127.0.98.62:38187)
0618 23:34:55.778271 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:55.778275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:55.778487 (+ 212us) server_negotiation.cc:407] Connection header received
0618 23:34:55.779575 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:55.779576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:55.779593 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:55.779686 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:55.781002 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.781830 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.783403 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.783717 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.783806 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:55.786300 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:55.786302 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:55.786302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:55.786312 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:55.788064 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:55.788564 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:55.788565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:55.788566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:55.788623 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:55.788891 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:55.788892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:55.788892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:55.789011 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:55.789079 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:55.789366 (+ 287us) server_negotiation.cc:299] Negotiation successful
0618 23:34:55.789468 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260618 23:34:55.789785 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:55.778071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.2:50955)
0618 23:34:55.778290 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:55.778354 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:55.779397 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:55.779845 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:55.779855 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:55.780453 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:55.780854 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:55.780863 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.781978 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.781979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:55.783303 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:55.783307 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.783846 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.783849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:55.784002 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:55.784590 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:55.784599 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:55.786178 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:55.788151 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:55.788158 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:55.788160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:55.788475 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:55.788701 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:55.788702 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:55.788702 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:55.788813 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:55.789083 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:55.789090 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:55.789237 (+ 147us) client_negotiation.cc:769] Sending connection context
0618 23:34:55.789348 (+ 111us) client_negotiation.cc:240] Negotiation successful
0618 23:34:55.789537 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":116,"threads_started":1}
I20260618 23:34:55.790385 16631 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38187
I20260618 23:34:55.790513 16631 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:55.790170 16523 raft_consensus.cc:348] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:34:55.790628 16523 raft_consensus.cc:729] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43363f29c3184a7d9f6f916bf560a182, State: Initialized, Role: FOLLOWER
I20260618 23:34:55.790709 16631 heartbeater.cc:511] Master 127.0.98.62:38187 requested a full tablet report, sending...
I20260618 23:34:55.790846 16523 consensus_queue.cc:260] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:34:55.791255 16523 ts_tablet_manager.cc:1428] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260618 23:34:55.791510 16304 ts_manager.cc:194] Registered new tserver with Master: 43363f29c3184a7d9f6f916bf560a182 (127.0.98.2:44357)
I20260618 23:34:55.792093 16304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50955
I20260618 23:34:55.793021 16631 heartbeater.cc:503] Master 127.0.98.62:38187 was elected leader, sending a full tablet report...
Jun 18 23:34:55 dist-test-slave-6224 krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260618 23:34:55.800494 16637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:55.792060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.0.98.2:44357)
0618 23:34:55.792300 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:55.792307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:55.792325 (+ 18us) server_negotiation.cc:407] Connection header received
0618 23:34:55.792377 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:55.792380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:55.792403 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:55.792547 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:55.792895 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.794122 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.794910 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.795393 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.795471 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:55.797519 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:55.797529 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:55.797529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:55.797539 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:55.799172 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:55.799677 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:55.799680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:55.799680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:55.799738 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:55.799955 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:55.799955 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:55.799956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:55.800098 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:55.800168 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:55.800329 (+ 161us) server_negotiation.cc:299] Negotiation successful
0618 23:34:55.800412 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":124,"threads_started":1}
I20260618 23:34:55.800555 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:55.791994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44357 (local address 127.0.0.1:50230)
0618 23:34:55.792235 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:55.792248 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:55.792299 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:55.792560 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:55.792561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:55.792738 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:55.792781 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:55.792783 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.794215 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.794216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:55.794820 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:55.794822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.795478 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.795481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:55.795528 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:55.796050 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:55.796055 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:55.797396 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:55.799292 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:55.799293 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:55.799294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:55.799575 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:55.799814 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:55.799814 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:55.799815 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:55.799886 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:55.800170 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:55.800171 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:55.800233 (+ 62us) client_negotiation.cc:769] Sending connection context
0618 23:34:55.800308 (+ 75us) client_negotiation.cc:240] Negotiation successful
0618 23:34:55.800439 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219}
I20260618 23:34:55.801014 16547 tablet_service.cc:1514] Processing DeleteTablet for tablet 95777b903b8b4b91a2108e4b547dd393 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:34:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50230
I20260618 23:34:55.801621 16638 tablet_replica.cc:330] stopping tablet replica
I20260618 23:34:55.801766 16638 raft_consensus.cc:2229] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:34:55.801849 16638 raft_consensus.cc:2258] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:34:55.802249 16638 ts_tablet_manager.cc:1905] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:34:55.804977 16638 ts_tablet_manager.cc:1918] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:34:55.805074 16638 log.cc:1196] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/95777b903b8b4b91a2108e4b547dd393
I20260618 23:34:55.805322 16638 ts_tablet_manager.cc:1939] T 95777b903b8b4b91a2108e4b547dd393 P 43363f29c3184a7d9f6f916bf560a182: Deleting consensus metadata
I20260618 23:34:55.806015 16291 catalog_manager.cc:4841] TS 43363f29c3184a7d9f6f916bf560a182 (127.0.98.2:44357): tablet 95777b903b8b4b91a2108e4b547dd393 (table test-table [id=6939179ec1cb4f84a5724a91fc15d4e8]) successfully deleted
W20260618 23:34:55.880472 16634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:34:55.880764 16634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:34:55.880795 16634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:34:55.882025 16634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:34:55.882066 16634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:34:55.882097 16634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:34:55.883651 16634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36389
--tserver_master_addrs=127.0.98.62:38187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:34:55.883884 16634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:34:55.884109 16634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:34:55.889905 16646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:55.889945 16643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:34:55.889905 16644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:34:55.890237 16634 server_base.cc:1034] running on GCE node
I20260618 23:34:55.890496 16634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:34:55.890744 16634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:34:55.891919 16634 hybrid_clock.cc:648] HybridClock initialized: now 1781825695891911 us; error 50 us; skew 500 ppm
Jun 18 23:34:55 dist-test-slave-6224 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:34:55.894547 16634 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:34:55.894996 16634 webserver.cc:460] Webserver started at http://127.0.98.3:36389/ using document root <none> and password file <none>
I20260618 23:34:55.895195 16634 fs_manager.cc:359] Metadata directory not provided
I20260618 23:34:55.895252 16634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:34:55.896785 16634 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:55.897470 16653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:55.897754 16634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:55.897863 16634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "d41f5d86f61f4ae99111fde71cace9dc"
format_stamp: "Formatted at 2026-06-18 23:34:35 on dist-test-slave-6224"
server_key: "3ca6241ff761b86c0910fcb4e7bff133"
server_key_iv: "7feacaba364bd1d8945d71174d0da67e"
server_key_version: "encryptionkey@0"
I20260618 23:34:55.898283 16634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:34:55.919061 16634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:34:55.921525 16634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:34:55.921677 16634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:34:55.921981 16634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:34:55.922317 16634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:34:55.922358 16634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:55.922380 16634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:34:55.922446 16634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:55.929517 16634 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44033
I20260618 23:34:55.929656 16766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44033 every 8 connection(s)
I20260618 23:34:55.929993 16634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
Jun 18 23:34:55 dist-test-slave-6224 krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:34:55.937069 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 16634
I20260618 23:34:55.941258 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:55.930870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39123 (local address 127.0.98.62:38187)
0618 23:34:55.930970 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:55.930973 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:55.931226 (+ 253us) server_negotiation.cc:407] Connection header received
0618 23:34:55.931840 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:55.931841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:55.931851 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:55.931947 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:55.933020 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.933515 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.934672 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.934863 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.934932 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:55.937683 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:55.937685 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:55.937685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:55.937693 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:55.939516 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:55.940183 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:55.940183 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:55.940184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:55.940237 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:55.940552 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:55.940552 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:55.940552 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:55.940651 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:55.940706 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:55.941070 (+ 364us) server_negotiation.cc:299] Negotiation successful
0618 23:34:55.941157 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 23:34:55.941884 16769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:55.930865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.3:39123)
0618 23:34:55.931060 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:55.931111 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:55.931731 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:55.932028 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:55.932032 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:55.932672 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:34:55.932926 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:55.932931 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.933650 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.933651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:55.934572 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:55.934574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.934971 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.934972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:55.935122 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:55.935762 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:34:55.935772 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:34:55.937528 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:34:55.939649 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:55.939658 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:55.939661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:55.940096 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:55.940318 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:34:55.940319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:34:55.940320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:34:55.940478 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:34:55.940726 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:34:55.940732 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:34:55.940927 (+ 195us) client_negotiation.cc:769] Sending connection context
0618 23:34:55.941033 (+ 106us) client_negotiation.cc:240] Negotiation successful
0618 23:34:55.941250 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":115,"threads_started":1}
I20260618 23:34:55.942561 16767 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38187
I20260618 23:34:55.942687 16767 heartbeater.cc:464] Registering TS with master...
I20260618 23:34:55.942935 16767 heartbeater.cc:511] Master 127.0.98.62:38187 requested a full tablet report, sending...
I20260618 23:34:55.943447 16304 ts_manager.cc:194] Registered new tserver with Master: d41f5d86f61f4ae99111fde71cace9dc (127.0.98.3:44033)
I20260618 23:34:55.943985 16304 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39123
I20260618 23:34:55.945766 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:55.937696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57366 (local address 127.0.98.62:38187)
0618 23:34:55.937890 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:55.937893 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:55.937905 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:34:55.937980 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:55.937981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:55.937993 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:55.938062 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:34:55.938903 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.939714 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.940839 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.941072 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.941146 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:55.942718 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:34:55.942721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:34:55.942721 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:34:55.942730 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:34:55.944610 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:55.945027 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:55.945029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:55.945029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:55.945094 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:34:55.945308 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:34:55.945309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:34:55.945310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:34:55.945434 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:34:55.945586 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:55.945623 (+ 37us) server_negotiation.cc:299] Negotiation successful
0618 23:34:55.945696 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":114,"threads_started":1}
I20260618 23:34:55.946650 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:34:55.949819 16771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:55.947043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57382 (local address 127.0.98.62:38187)
0618 23:34:55.947119 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:55.947121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:55.947264 (+ 143us) server_negotiation.cc:407] Connection header received
0618 23:34:55.947306 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:55.947306 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:55.947335 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:55.947399 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:34:55.947525 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.948063 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.948913 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.949113 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.949173 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:55.949418 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:34:55.949514 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:34:55.949605 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:55.949704 (+ 99us) server_negotiation.cc:299] Negotiation successful
0618 23:34:55.949725 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 23:34:55.950271 16304 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:34:55.952124 16304 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:34:55.954787 16547 tablet_service.cc:1467] Processing CreateTablet for tablet 1890ceeeeaca4588b3769893f5ec639c (DEFAULT_TABLE table=test-table [id=ea570a7c1f3d42459cd32d3527bccb3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:34:55.954885 16426 tablet_service.cc:1467] Processing CreateTablet for tablet 1890ceeeeaca4588b3769893f5ec639c (DEFAULT_TABLE table=test-table [id=ea570a7c1f3d42459cd32d3527bccb3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:34:55.955014 16547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1890ceeeeaca4588b3769893f5ec639c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:34:55.955034 16426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1890ceeeeaca4588b3769893f5ec639c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:34:55.956652 16523 tablet_bootstrap.cc:492] T 1890ceeeeaca4588b3769893f5ec639c P 43363f29c3184a7d9f6f916bf560a182: Bootstrap starting.
I20260618 23:34:55.956652 16384 tablet_bootstrap.cc:492] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb: Bootstrap starting.
I20260618 23:34:55.956979 16523 tablet_bootstrap.cc:654] T 1890ceeeeaca4588b3769893f5ec639c P 43363f29c3184a7d9f6f916bf560a182: Neither blocks nor log segments found. Creating new log.
I20260618 23:34:55.956979 16384 tablet_bootstrap.cc:654] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb: Neither blocks nor log segments found. Creating new log.
I20260618 23:34:55.957687 16384 tablet_bootstrap.cc:492] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb: No bootstrap required, opened a new log
I20260618 23:34:55.957738 16384 ts_tablet_manager.cc:1397] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:55.958027 16523 tablet_bootstrap.cc:492] T 1890ceeeeaca4588b3769893f5ec639c P 43363f29c3184a7d9f6f916bf560a182: No bootstrap required, opened a new log
I20260618 23:34:55.958069 16523 ts_tablet_manager.cc:1397] T 1890ceeeeaca4588b3769893f5ec639c P 43363f29c3184a7d9f6f916bf560a182: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260618 23:34:55.958113 16384 raft_consensus.cc:348] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:34:55.958189 16384 raft_consensus.cc:374] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:34:55.958231 16384 raft_consensus.cc:729] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6e88a6a1404de09e8beaf85d5d4deb, State: Initialized, Role: FOLLOWER
I20260618 23:34:55.958202 16523 raft_consensus.cc:348] T 1890ceeeeaca4588b3769893f5ec639c P 43363f29c3184a7d9f6f916bf560a182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:34:55.958263 16523 raft_consensus.cc:374] T 1890ceeeeaca4588b3769893f5ec639c P 43363f29c3184a7d9f6f916bf560a182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:34:55.958302 16523 raft_consensus.cc:729] T 1890ceeeeaca4588b3769893f5ec639c P 43363f29c3184a7d9f6f916bf560a182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43363f29c3184a7d9f6f916bf560a182, State: Initialized, Role: FOLLOWER
I20260618 23:34:55.958293 16384 consensus_queue.cc:260] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:34:55.958348 16523 consensus_queue.cc:260] T 1890ceeeeaca4588b3769893f5ec639c P 43363f29c3184a7d9f6f916bf560a182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:34:55.958400 16384 ts_tablet_manager.cc:1428] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:34:55.958441 16523 ts_tablet_manager.cc:1428] T 1890ceeeeaca4588b3769893f5ec639c P 43363f29c3184a7d9f6f916bf560a182: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:34:55.958787 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:55.954568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44033 (local address 127.0.0.1:42798)
0618 23:34:55.954614 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:55.954628 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:55.954674 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:55.955132 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:55.955133 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:55.955136 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:34:55.955201 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:55.955203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.956533 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.956534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:55.957549 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:55.957553 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.958494 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.958495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:55.958558 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:55.958576 (+ 18us) client_negotiation.cc:769] Sending connection context
0618 23:34:55.958630 (+ 54us) client_negotiation.cc:240] Negotiation successful
0618 23:34:55.958686 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260618 23:34:55.959085 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:55.954669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.0.98.3:44033)
0618 23:34:55.954893 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:55.954899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:55.954912 (+ 13us) server_negotiation.cc:407] Connection header received
0618 23:34:55.954961 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:55.954964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:55.954983 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:55.955337 (+ 354us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:34:55.955438 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.956439 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.957659 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.958264 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.958361 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:55.958430 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:55.958904 (+ 474us) server_negotiation.cc:299] Negotiation successful
0618 23:34:55.958977 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":146,"threads_started":1}
I20260618 23:34:55.960005 16701 tablet_service.cc:1467] Processing CreateTablet for tablet 1890ceeeeaca4588b3769893f5ec639c (DEFAULT_TABLE table=test-table [id=ea570a7c1f3d42459cd32d3527bccb3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:34:55.960418 16701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1890ceeeeaca4588b3769893f5ec639c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:34:55.963778 16776 tablet_bootstrap.cc:492] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc: Bootstrap starting.
I20260618 23:34:55.964481 16776 tablet_bootstrap.cc:654] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc: Neither blocks nor log segments found. Creating new log.
I20260618 23:34:55.965106 16776 log.cc:824] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc: Log is configured to *not* fsync() on all Append() calls
I20260618 23:34:55.965943 16776 tablet_bootstrap.cc:492] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc: No bootstrap required, opened a new log
I20260618 23:34:55.966008 16776 ts_tablet_manager.cc:1397] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260618 23:34:55.967605 16776 raft_consensus.cc:348] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:34:55.967778 16776 raft_consensus.cc:374] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:34:55.967803 16776 raft_consensus.cc:729] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d41f5d86f61f4ae99111fde71cace9dc, State: Initialized, Role: FOLLOWER
I20260618 23:34:55.967952 16776 consensus_queue.cc:260] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:34:55.968251 16776 ts_tablet_manager.cc:1428] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:34:55.968400 16767 heartbeater.cc:503] Master 127.0.98.62:38187 was elected leader, sending a full tablet report...
I20260618 23:34:55.969177 16778 raft_consensus.cc:482] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:34:55.969338 16778 raft_consensus.cc:504] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:34:55.969707 16778 leader_election.cc:288] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43363f29c3184a7d9f6f916bf560a182 (127.0.98.2:44357), aa6e88a6a1404de09e8beaf85d5d4deb (127.0.98.1:38713)
I20260618 23:34:55.974261 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:55.970004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34517 (local address 127.0.98.1:38713)
0618 23:34:55.970068 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:55.970070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:55.970190 (+ 120us) server_negotiation.cc:407] Connection header received
0618 23:34:55.970267 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:55.970268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:55.970296 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:55.970374 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:34:55.970608 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.971387 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.973144 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.973831 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.973922 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:55.973989 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:55.974143 (+ 154us) server_negotiation.cc:299] Negotiation successful
0618 23:34:55.974186 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260618 23:34:55.974274 16637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:55.969942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47587 (local address 127.0.98.2:44357)
0618 23:34:55.969986 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:55.969988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:55.970202 (+ 214us) server_negotiation.cc:407] Connection header received
0618 23:34:55.970268 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:55.970269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:55.970296 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:55.970374 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:34:55.970606 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.971384 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.973145 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.973831 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.973922 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:55.973989 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:55.974143 (+ 154us) server_negotiation.cc:299] Negotiation successful
0618 23:34:55.974187 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260618 23:34:55.974288 16769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:55.969957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38713 (local address 127.0.98.3:34517)
0618 23:34:55.970111 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:55.970125 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:55.970187 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:55.970394 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:55.970395 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:55.970398 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:34:55.970526 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:55.970529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.971505 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.971506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:55.973019 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:55.973023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.973933 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.973934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:55.974031 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:55.974037 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:34:55.974109 (+ 72us) client_negotiation.cc:240] Negotiation successful
0618 23:34:55.974179 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":155,"thread_start_us":104,"threads_started":1}
I20260618 23:34:55.974296 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:55.969826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44357 (local address 127.0.98.3:47587)
0618 23:34:55.970111 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:55.970125 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:34:55.970196 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:34:55.970394 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:34:55.970395 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:34:55.970398 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:34:55.970528 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:55.970529 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.971513 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.971514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:55.973019 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:34:55.973023 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.973934 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.973934 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:34:55.974059 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:55.974063 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:34:55.974122 (+ 59us) client_negotiation.cc:240] Negotiation successful
0618 23:34:55.974179 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":80}
I20260618 23:34:55.974654 16446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1890ceeeeaca4588b3769893f5ec639c" candidate_uuid: "d41f5d86f61f4ae99111fde71cace9dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" is_pre_election: true
I20260618 23:34:55.974848 16446 raft_consensus.cc:2454] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d41f5d86f61f4ae99111fde71cace9dc in term 0.
I20260618 23:34:55.974891 16585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1890ceeeeaca4588b3769893f5ec639c" candidate_uuid: "d41f5d86f61f4ae99111fde71cace9dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43363f29c3184a7d9f6f916bf560a182" is_pre_election: true
I20260618 23:34:55.975029 16585 raft_consensus.cc:2454] T 1890ceeeeaca4588b3769893f5ec639c P 43363f29c3184a7d9f6f916bf560a182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d41f5d86f61f4ae99111fde71cace9dc in term 0.
I20260618 23:34:55.975088 16656 leader_election.cc:302] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa6e88a6a1404de09e8beaf85d5d4deb, d41f5d86f61f4ae99111fde71cace9dc; no voters:
I20260618 23:34:55.975221 16778 raft_consensus.cc:2781] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:34:55.975278 16778 raft_consensus.cc:482] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:34:55.975317 16778 raft_consensus.cc:3037] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:34:55.976202 16778 raft_consensus.cc:504] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:34:55.976375 16778 leader_election.cc:288] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc [CANDIDATE]: Term 1 election: Requested vote from peers 43363f29c3184a7d9f6f916bf560a182 (127.0.98.2:44357), aa6e88a6a1404de09e8beaf85d5d4deb (127.0.98.1:38713)
I20260618 23:34:55.976522 16446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1890ceeeeaca4588b3769893f5ec639c" candidate_uuid: "d41f5d86f61f4ae99111fde71cace9dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb"
I20260618 23:34:55.976562 16585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1890ceeeeaca4588b3769893f5ec639c" candidate_uuid: "d41f5d86f61f4ae99111fde71cace9dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43363f29c3184a7d9f6f916bf560a182"
I20260618 23:34:55.976627 16446 raft_consensus.cc:3037] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:34:55.976653 16585 raft_consensus.cc:3037] T 1890ceeeeaca4588b3769893f5ec639c P 43363f29c3184a7d9f6f916bf560a182 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:34:55.977437 16446 raft_consensus.cc:2454] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d41f5d86f61f4ae99111fde71cace9dc in term 1.
I20260618 23:34:55.977445 16585 raft_consensus.cc:2454] T 1890ceeeeaca4588b3769893f5ec639c P 43363f29c3184a7d9f6f916bf560a182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d41f5d86f61f4ae99111fde71cace9dc in term 1.
I20260618 23:34:55.977662 16656 leader_election.cc:302] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa6e88a6a1404de09e8beaf85d5d4deb, d41f5d86f61f4ae99111fde71cace9dc; no voters:
I20260618 23:34:55.977777 16778 raft_consensus.cc:2781] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:34:55.977972 16778 raft_consensus.cc:686] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc [term 1 LEADER]: Becoming Leader. State: Replica: d41f5d86f61f4ae99111fde71cace9dc, State: Running, Role: LEADER
I20260618 23:34:55.978088 16778 consensus_queue.cc:237] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:34:55.979091 16304 catalog_manager.cc:5495] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc reported cstate change: term changed from 0 to 1, leader changed from <none> to d41f5d86f61f4ae99111fde71cace9dc (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d41f5d86f61f4ae99111fde71cace9dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } health_report { overall_health: UNKNOWN } } }
I20260618 23:34:55.989925 16773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:55.987390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42814 (local address 127.0.98.3:44033)
0618 23:34:55.987443 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0618 23:34:55.987445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:34:55.987453 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:34:55.987626 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:34:55.987627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:34:55.987654 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:34:55.987737 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:34:55.987859 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.988362 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.989108 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:34:55.989303 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:34:55.989362 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:34:55.989569 (+ 207us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:34:55.989685 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:34:55.989757 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 23:34:55.989801 (+ 44us) server_negotiation.cc:299] Negotiation successful
0618 23:34:55.989825 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260618 23:34:55.992259 16446 raft_consensus.cc:1261] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Refusing update from remote peer d41f5d86f61f4ae99111fde71cace9dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:34:55.992316 16585 raft_consensus.cc:1261] T 1890ceeeeaca4588b3769893f5ec639c P 43363f29c3184a7d9f6f916bf560a182 [term 1 FOLLOWER]: Refusing update from remote peer d41f5d86f61f4ae99111fde71cace9dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:34:55.992765 16778 consensus_queue.cc:1035] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:34:55.992971 16780 consensus_queue.cc:1035] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:34:55.995226 16783 mvcc.cc:205] Tried to move back new op lower bound from 7298358050780598272 to 7298358050727268352. Current Snapshot: MvccSnapshot[applied={T|T < 7298358050780598272}]
I20260618 23:34:55.995764 16782 mvcc.cc:205] Tried to move back new op lower bound from 7298358050780598272 to 7298358050727268352. Current Snapshot: MvccSnapshot[applied={T|T < 7298358050780598272 or (T in {7298358050780598272})}]
I20260618 23:34:55.997825 16304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:34:55.997925 16304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:34:55.998806 16304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b: Sending DeleteTablet for 3 replicas of tablet 1890ceeeeaca4588b3769893f5ec639c
I20260618 23:34:55.999172 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 16342
I20260618 23:34:55.999276 16426 tablet_service.cc:1514] Processing DeleteTablet for tablet 1890ceeeeaca4588b3769893f5ec639c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:34:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33386
I20260618 23:34:55.999274 16547 tablet_service.cc:1514] Processing DeleteTablet for tablet 1890ceeeeaca4588b3769893f5ec639c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:34:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50230
I20260618 23:34:55.999432 16701 tablet_service.cc:1514] Processing DeleteTablet for tablet 1890ceeeeaca4588b3769893f5ec639c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:34:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42798
I20260618 23:34:55.999696 16638 tablet_replica.cc:330] T 1890ceeeeaca4588b3769893f5ec639c P 43363f29c3184a7d9f6f916bf560a182: stopping tablet replica
I20260618 23:34:55.999766 16638 raft_consensus.cc:2229] T 1890ceeeeaca4588b3769893f5ec639c P 43363f29c3184a7d9f6f916bf560a182 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:34:55.999841 16638 raft_consensus.cc:2258] T 1890ceeeeaca4588b3769893f5ec639c P 43363f29c3184a7d9f6f916bf560a182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:34:56.000072 16792 tablet_replica.cc:330] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc: stopping tablet replica
I20260618 23:34:56.000183 16792 raft_consensus.cc:2229] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:34:56.000258 16638 ts_tablet_manager.cc:1905] T 1890ceeeeaca4588b3769893f5ec639c P 43363f29c3184a7d9f6f916bf560a182: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:34:56.000289 16792 raft_consensus.cc:2258] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:34:56.000888 16792 ts_tablet_manager.cc:1905] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:34:56.002250 16638 ts_tablet_manager.cc:1918] T 1890ceeeeaca4588b3769893f5ec639c P 43363f29c3184a7d9f6f916bf560a182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:34:56.002310 16638 log.cc:1196] T 1890ceeeeaca4588b3769893f5ec639c P 43363f29c3184a7d9f6f916bf560a182: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/1890ceeeeaca4588b3769893f5ec639c
I20260618 23:34:56.002533 16638 ts_tablet_manager.cc:1939] T 1890ceeeeaca4588b3769893f5ec639c P 43363f29c3184a7d9f6f916bf560a182: Deleting consensus metadata
I20260618 23:34:56.002938 16792 ts_tablet_manager.cc:1918] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:34:56.003021 16792 log.cc:1196] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/1890ceeeeaca4588b3769893f5ec639c
I20260618 23:34:56.003072 16291 catalog_manager.cc:4841] TS 43363f29c3184a7d9f6f916bf560a182 (127.0.98.2:44357): tablet 1890ceeeeaca4588b3769893f5ec639c (table test-table [id=ea570a7c1f3d42459cd32d3527bccb3d]) successfully deleted
I20260618 23:34:56.003336 16792 ts_tablet_manager.cc:1939] T 1890ceeeeaca4588b3769893f5ec639c P d41f5d86f61f4ae99111fde71cace9dc: Deleting consensus metadata
I20260618 23:34:56.003860 16290 catalog_manager.cc:4841] TS d41f5d86f61f4ae99111fde71cace9dc (127.0.98.3:44033): tablet 1890ceeeeaca4588b3769893f5ec639c (table test-table [id=ea570a7c1f3d42459cd32d3527bccb3d]) successfully deleted
W20260618 23:34:56.006500 16290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38713 (error 108)
I20260618 23:34:56.006816 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 16497
I20260618 23:34:56.009330 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:34:56.009176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38713 (local address 127.0.0.1:33398)
0618 23:34:56.009248 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0618 23:34:56.009294 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260618 23:34:56.009474 16290 catalog_manager.cc:4584] TS aa6e88a6a1404de09e8beaf85d5d4deb (127.0.98.1:38713): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1890ceeeeaca4588b3769893f5ec639c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38713: connect: Connection refused (error 111)
I20260618 23:34:56.013785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 16634
I20260618 23:34:56.019519 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 16271
I20260618 23:35:16.026118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38187
--webserver_interface=127.0.98.62
--webserver_port=35687
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:16.120673 16794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:16.120952 16794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:16.120980 16794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:16.122370 16794 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260618 23:35:16.122421 16794 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260618 23:35:16.122434 16794 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260618 23:35:16.122447 16794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:35:16.122462 16794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:16.122475 16794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:16.122488 16794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:35:16.122500 16794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:35:16.124122 16794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:16.124373 16794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:16.124594 16794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:16.131183 16799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:16.131323 16794 server_base.cc:1034] running on GCE node
W20260618 23:35:16.131251 16800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:16.131251 16802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:16.131938 16794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:16.132356 16794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:16.133534 16794 hybrid_clock.cc:648] HybridClock initialized: now 1781825716133503 us; error 56 us; skew 500 ppm
Jun 18 23:35:16 dist-test-slave-6224 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:16.142412 16794 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:35:16.143050 16794 webserver.cc:460] Webserver started at http://127.0.98.62:35687/ using document root <none> and password file <none>
I20260618 23:35:16.143347 16794 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:16.143421 16794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:16.145184 16794 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:16.146137 16809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:16.146487 16794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:16.146636 16794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "85670a759a204c2ab74dae692caffc5b"
format_stamp: "Formatted at 2026-06-18 23:34:34 on dist-test-slave-6224"
server_key: "bc5c96dfa5a9eaa052704edb3d5c2809"
server_key_iv: "3e0e83ad73c466fba07f8a0aabe630a9"
server_key_version: "encryptionkey@0"
I20260618 23:35:16.147090 16794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:16.162981 16794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:16.165486 16794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:16.165669 16794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:16.170893 16794 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38187
I20260618 23:35:16.170965 16861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38187 every 8 connection(s)
I20260618 23:35:16.171420 16794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:35:16.172923 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 16794
I20260618 23:35:16.173230 16862 sys_catalog.cc:263] Verifying existing consensus state
I20260618 23:35:16.173300 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38713
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38187
--webserver_port=38985
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:35:16.173993 16862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b: Bootstrap starting.
I20260618 23:35:16.183535 16862 log.cc:824] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:16.186022 16862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:35:16.186372 16862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b: Bootstrap complete.
I20260618 23:35:16.188794 16862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85670a759a204c2ab74dae692caffc5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38187 } }
I20260618 23:35:16.189154 16862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85670a759a204c2ab74dae692caffc5b, State: Initialized, Role: FOLLOWER
I20260618 23:35:16.189378 16862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85670a759a204c2ab74dae692caffc5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38187 } }
I20260618 23:35:16.189484 16862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:35:16.189538 16862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:35:16.189589 16862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 2 FOLLOWER]: Advancing to term 3
I20260618 23:35:16.190645 16862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85670a759a204c2ab74dae692caffc5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38187 } }
I20260618 23:35:16.190771 16862 leader_election.cc:302] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85670a759a204c2ab74dae692caffc5b; no voters:
I20260618 23:35:16.190985 16862 leader_election.cc:288] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260618 23:35:16.191107 16866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 3 FOLLOWER]: Leader election won for term 3
I20260618 23:35:16.191306 16862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:35:16.191468 16866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [term 3 LEADER]: Becoming Leader. State: Replica: 85670a759a204c2ab74dae692caffc5b, State: Running, Role: LEADER
I20260618 23:35:16.191717 16866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85670a759a204c2ab74dae692caffc5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38187 } }
I20260618 23:35:16.192353 16867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "85670a759a204c2ab74dae692caffc5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85670a759a204c2ab74dae692caffc5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38187 } } }
I20260618 23:35:16.192489 16867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:16.192854 16868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85670a759a204c2ab74dae692caffc5b. Latest consensus state: current_term: 3 leader_uuid: "85670a759a204c2ab74dae692caffc5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85670a759a204c2ab74dae692caffc5b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38187 } } }
I20260618 23:35:16.192946 16868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:16.192889 16879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:35:16.193727 16879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:35:16.193948 16879 catalog_manager.cc:1259] Loaded cluster ID: 2b34a157759b4477acbff5f2ae262ed3
I20260618 23:35:16.193989 16879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:35:16.194898 16879 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:35:16.195107 16879 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b: Loaded TSK: 0
I20260618 23:35:16.195149 16879 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85670a759a204c2ab74dae692caffc5b: Loaded TSK: 1
F20260618 23:35:16.195396 16879 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe9816b59ad google::LogMessage::Fail() at ??:0
@ 0x7fe9816b888a google::LogMessage::SendToLog() at ??:0
@ 0x7fe9816b54de google::LogMessage::Flush() at ??:0
@ 0x7fe9816b6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe983e22104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe983e1fd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe98960f1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe98960f48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe98960f594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe9895d5c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fe9895dfad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe9835830d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe983578954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe9843dc6db start_thread at ??:0
@ 0x7fe9801b361f clone at ??:0
W20260618 23:35:16.281502 16864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:16.281781 16864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:16.281821 16864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:16.283013 16864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:16.283084 16864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:16.283133 16864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:35:16.284755 16864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38713
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38985
--tserver_master_addrs=127.0.98.62:38187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:16.285059 16864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:16.285323 16864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:16.291157 16904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:16.291306 16903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:16.291617 16906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:16.292114 16864 server_base.cc:1034] running on GCE node
I20260618 23:35:16.292349 16864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:16.292671 16864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:16.296038 16864 hybrid_clock.cc:648] HybridClock initialized: now 1781825716295986 us; error 27 us; skew 500 ppm
Jun 18 23:35:16 dist-test-slave-6224 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:16.300839 16864 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:35:16.301749 16864 webserver.cc:460] Webserver started at http://127.0.98.1:38985/ using document root <none> and password file <none>
I20260618 23:35:16.302098 16864 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:16.302186 16864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:16.304965 16864 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260618 23:35:16.306489 16913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:16.306988 16864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:16.307097 16864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "aa6e88a6a1404de09e8beaf85d5d4deb"
format_stamp: "Formatted at 2026-06-18 23:34:34 on dist-test-slave-6224"
server_key: "b7dacc4ec7279f24a11b02643a1b9f2b"
server_key_iv: "8c3a669a24149b8364408a56a0ec853f"
server_key_version: "encryptionkey@0"
I20260618 23:35:16.307615 16864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:16.335242 16864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:16.338827 16864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:16.339025 16864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:16.339373 16864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:35:16.340111 16920 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260618 23:35:16.342510 16864 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 23:35:16.342664 16864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260618 23:35:16.342823 16864 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 23:35:16.344221 16864 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 23:35:16.344364 16864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260618 23:35:16.344391 16920 tablet_bootstrap.cc:492] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb: Bootstrap starting.
I20260618 23:35:16.354281 16864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38713
I20260618 23:35:16.354765 16864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:35:16.355489 17027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38713 every 8 connection(s)
I20260618 23:35:16.360608 16920 log.cc:824] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:16.360808 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 16864
I20260618 23:35:16.361217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:44357
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38187
--webserver_port=38369
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:35:16.364037 16920 tablet_bootstrap.cc:492] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:35:16.364447 16920 tablet_bootstrap.cc:492] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb: Bootstrap complete.
I20260618 23:35:16.365743 16920 ts_tablet_manager.cc:1397] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.012s
I20260618 23:35:16.367285 16920 raft_consensus.cc:348] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:35:16.367751 16920 raft_consensus.cc:729] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa6e88a6a1404de09e8beaf85d5d4deb, State: Initialized, Role: FOLLOWER
I20260618 23:35:16.367935 16920 consensus_queue.cc:260] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:35:16.368299 16920 ts_tablet_manager.cc:1428] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260618 23:35:16.426713 17030 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:35:16.355658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.1:49653)
0618 23:35:16.355868 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:16.355919 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:16.356513 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:16.426463 (+ 69950us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":123,"threads_started":1}
W20260618 23:35:16.427091 17028 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38187 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38187: Client connection negotiation failed: client connection to 127.0.98.62:38187: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260618 23:35:16.427525 17030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:16.427388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.1:39567)
0618 23:35:16.427449 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:16.427499 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260618 23:35:16.427911 17030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:16.427824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.1:43709)
0618 23:35:16.427867 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:16.427891 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260618 23:35:16.428082 17028 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 23:35:16.458739 17032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:16.459043 17032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:16.459091 17032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:16.460559 17032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:16.460645 17032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:16.460696 17032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:35:16.462358 17032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:44357
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38369
--tserver_master_addrs=127.0.98.62:38187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:16.462674 17032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:16.462960 17032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:16.468906 17041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:16.468917 17038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:16.468967 17039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:16.469030 17032 server_base.cc:1034] running on GCE node
I20260618 23:35:16.469508 17032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:16.469815 17032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:16.471076 17032 hybrid_clock.cc:648] HybridClock initialized: now 1781825716470991 us; error 75 us; skew 500 ppm
Jun 18 23:35:16 dist-test-slave-6224 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:16.474504 17032 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:35:16.475106 17032 webserver.cc:460] Webserver started at http://127.0.98.2:38369/ using document root <none> and password file <none>
I20260618 23:35:16.475378 17032 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:16.475454 17032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:16.477162 17032 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:35:16.478004 17048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:16.478376 17032 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:35:16.478495 17032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "43363f29c3184a7d9f6f916bf560a182"
format_stamp: "Formatted at 2026-06-18 23:34:34 on dist-test-slave-6224"
server_key: "81708018309be01e04845b8534047d5b"
server_key_iv: "0271e2adcae4c10e6cbef54c0b366804"
server_key_version: "encryptionkey@0"
I20260618 23:35:16.478953 17032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:16.490489 17032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:16.492966 17032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:16.493145 17032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:16.493432 17032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:35:16.493798 17032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:35:16.493853 17032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:16.493903 17032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:35:16.493928 17032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:16.501698 17032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44357
I20260618 23:35:16.501749 17161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44357 every 8 connection(s)
I20260618 23:35:16.502206 17032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:35:16.503521 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:16.503019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.2:55163)
0618 23:35:16.503246 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:16.503352 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":93,"threads_started":1}
W20260618 23:35:16.504076 17162 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38187 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38187: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
I20260618 23:35:16.504498 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:16.504359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.2:42141)
0618 23:35:16.504435 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:16.504474 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260618 23:35:16.504894 17164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:16.504787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.2:38795)
0618 23:35:16.504849 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:16.504877 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260618 23:35:16.505074 17162 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260618 23:35:16.507823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 17032
I20260618 23:35:16.508112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44033
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38187
--webserver_port=36389
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:16.595949 17165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:16.596185 17165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:16.596212 17165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:16.597330 17165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:16.597374 17165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:16.597401 17165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:35:16.598863 17165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36389
--tserver_master_addrs=127.0.98.62:38187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:16.599062 17165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:16.599246 17165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:16.605207 17170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:16.605214 17173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:16.605204 17171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:16.605686 17165 server_base.cc:1034] running on GCE node
I20260618 23:35:16.605924 17165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:16.606222 17165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:16.607380 17165 hybrid_clock.cc:648] HybridClock initialized: now 1781825716607355 us; error 50 us; skew 500 ppm
Jun 18 23:35:16 dist-test-slave-6224 krb5kdc[15724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:16.610383 17165 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:35:16.611022 17165 webserver.cc:460] Webserver started at http://127.0.98.3:36389/ using document root <none> and password file <none>
I20260618 23:35:16.611289 17165 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:16.611338 17165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:16.613231 17165 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:35:16.614007 17180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:16.614348 17165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:35:16.614466 17165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "d41f5d86f61f4ae99111fde71cace9dc"
format_stamp: "Formatted at 2026-06-18 23:34:35 on dist-test-slave-6224"
server_key: "3ca6241ff761b86c0910fcb4e7bff133"
server_key_iv: "7feacaba364bd1d8945d71174d0da67e"
server_key_version: "encryptionkey@0"
I20260618 23:35:16.614909 17165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:16.638917 17165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:16.641227 17165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:16.641424 17165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:16.641743 17165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:35:16.642112 17165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:35:16.642171 17165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:16.642216 17165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:35:16.642258 17165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:16.649995 17165 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44033
I20260618 23:35:16.650060 17293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44033 every 8 connection(s)
I20260618 23:35:16.650511 17165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:35:16.651859 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:16.651354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.3:56399)
0618 23:35:16.651592 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:16.651702 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":86,"threads_started":1}
W20260618 23:35:16.652377 17294 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38187 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38187: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
I20260618 23:35:16.652750 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:16.652647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.3:36931)
0618 23:35:16.652700 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:16.652732 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260618 23:35:16.653095 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:16.653010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.3:35007)
0618 23:35:16.653056 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:16.653081 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260618 23:35:16.653241 17294 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260618 23:35:16.653949 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 17165
I20260618 23:35:17.428784 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:17.428441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.1:34451)
0618 23:35:17.428639 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:17.428690 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":67,"threads_started":1}
I20260618 23:35:17.505754 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:17.505463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.2:57569)
0618 23:35:17.505627 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:17.505675 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":64,"threads_started":1}
I20260618 23:35:17.653957 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:17.653662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.3:60839)
0618 23:35:17.653819 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:17.653868 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
I20260618 23:35:17.895829 17301 raft_consensus.cc:482] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:35:17.895972 17301 raft_consensus.cc:504] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:35:17.896396 17301 leader_election.cc:288] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d41f5d86f61f4ae99111fde71cace9dc (127.0.98.3:44033), 43363f29c3184a7d9f6f916bf560a182 (127.0.98.2:44357)
Jun 18 23:35:17 dist-test-slave-6224 krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 18 23:35:17 dist-test-slave-6224 krb5kdc[15724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825716, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260618 23:35:17.910734 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:17.896644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44033 (local address 127.0.98.1:59613)
0618 23:35:17.896832 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:17.896845 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:17.896919 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:17.897976 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:17.897980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:17.898659 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:35:17.899009 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:17.899019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:17.900762 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:17.900763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:17.902159 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:17.902162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:17.903068 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:17.903069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:17.903238 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:17.904076 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:35:17.904088 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:35:17.906135 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:35:17.908400 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:35:17.908408 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:35:17.908411 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:35:17.908743 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:35:17.909117 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:35:17.909118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:35:17.909119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:35:17.909298 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:35:17.909977 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:35:17.909986 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:35:17.910170 (+ 184us) client_negotiation.cc:769] Sending connection context
0618 23:35:17.910349 (+ 179us) client_negotiation.cc:240] Negotiation successful
0618 23:35:17.910607 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":109,"thread_start_us":83,"threads_started":1}
I20260618 23:35:17.910770 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:17.896839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59613 (local address 127.0.98.3:44033)
0618 23:35:17.897037 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:17.897044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:17.897083 (+ 39us) server_negotiation.cc:407] Connection header received
0618 23:35:17.897635 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:17.897638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:17.897695 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:17.897873 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:17.899158 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:17.900625 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:17.902287 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:17.902953 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:17.903077 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:17.906300 (+ 3223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:17.906305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:17.906305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:17.906328 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:17.908302 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:17.908844 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:17.908847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:17.908847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:17.909032 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:17.909401 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:17.909403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:17.909403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:17.909840 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:17.909938 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:17.910414 (+ 476us) server_negotiation.cc:299] Negotiation successful
0618 23:35:17.910639 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":80,"threads_started":1}
I20260618 23:35:17.910857 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:17.896734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47057 (local address 127.0.98.2:44357)
0618 23:35:17.896914 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:17.896945 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:17.896994 (+ 49us) server_negotiation.cc:407] Connection header received
0618 23:35:17.897635 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:17.897638 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:17.897695 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:17.897873 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:17.899158 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:17.900630 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:17.902287 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:17.902952 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:17.903077 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:17.906750 (+ 3673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:17.906755 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:17.906756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:17.906771 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:17.908905 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:17.909536 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:17.909539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:17.909539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:17.909639 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:17.909863 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:17.909864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:17.909864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:17.910191 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:17.910295 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:17.910573 (+ 278us) server_negotiation.cc:299] Negotiation successful
0618 23:35:17.910723 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":75,"threads_started":1}
I20260618 23:35:17.911684 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:17.896603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44357 (local address 127.0.98.1:47057)
0618 23:35:17.896668 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:17.896682 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:17.896739 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:17.897976 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:17.897980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:17.898659 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:35:17.899009 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:17.899019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:17.900753 (+ 1734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:17.900754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:17.902159 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:17.902162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:17.903070 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:17.903070 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:17.903238 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:17.904100 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:35:17.904105 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:35:17.906579 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:35:17.909222 (+ 2643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:35:17.909223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:35:17.909223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:35:17.909438 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:35:17.909717 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:35:17.909718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:35:17.909718 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:35:17.909785 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:35:17.910303 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:35:17.910304 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:35:17.910405 (+ 101us) client_negotiation.cc:769] Sending connection context
0618 23:35:17.910494 (+ 89us) client_negotiation.cc:240] Negotiation successful
0618 23:35:17.910699 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":169}
W20260618 23:35:17.911998 16916 leader_election.cc:341] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d41f5d86f61f4ae99111fde71cace9dc (127.0.98.3:44033): Not found: Tablet not found: 1890ceeeeaca4588b3769893f5ec639c
W20260618 23:35:17.912449 16917 leader_election.cc:341] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43363f29c3184a7d9f6f916bf560a182 (127.0.98.2:44357): Not found: Tablet not found: 1890ceeeeaca4588b3769893f5ec639c
I20260618 23:35:17.912510 16917 leader_election.cc:302] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa6e88a6a1404de09e8beaf85d5d4deb; no voters: 43363f29c3184a7d9f6f916bf560a182, d41f5d86f61f4ae99111fde71cace9dc
I20260618 23:35:17.912680 17301 raft_consensus.cc:2726] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 23:35:18.429780 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:18.429462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.1:58755)
0618 23:35:18.429639 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:18.429715 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":73,"threads_started":1}
I20260618 23:35:18.506712 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:18.506412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.2:60663)
0618 23:35:18.506583 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:18.506650 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":75,"threads_started":1}
I20260618 23:35:18.654995 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:18.654674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.3:40883)
0618 23:35:18.654881 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:18.654950 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":88,"threads_started":1}
I20260618 23:35:19.430759 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:19.430424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.1:47843)
0618 23:35:19.430624 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:19.430693 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":82,"threads_started":1}
I20260618 23:35:19.507712 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:19.507327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.2:32973)
0618 23:35:19.507503 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:19.507632 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":73,"threads_started":1}
I20260618 23:35:19.656119 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:19.655862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.3:42667)
0618 23:35:19.656029 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:19.656071 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":81,"threads_started":1}
I20260618 23:35:19.671849 17311 raft_consensus.cc:482] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:35:19.671972 17311 raft_consensus.cc:504] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:35:19.672154 17311 leader_election.cc:288] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d41f5d86f61f4ae99111fde71cace9dc (127.0.98.3:44033), 43363f29c3184a7d9f6f916bf560a182 (127.0.98.2:44357)
W20260618 23:35:19.672677 16917 leader_election.cc:341] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43363f29c3184a7d9f6f916bf560a182 (127.0.98.2:44357): Not found: Tablet not found: 1890ceeeeaca4588b3769893f5ec639c
W20260618 23:35:19.672786 16916 leader_election.cc:341] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d41f5d86f61f4ae99111fde71cace9dc (127.0.98.3:44033): Not found: Tablet not found: 1890ceeeeaca4588b3769893f5ec639c
I20260618 23:35:19.672912 16916 leader_election.cc:302] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa6e88a6a1404de09e8beaf85d5d4deb; no voters: 43363f29c3184a7d9f6f916bf560a182, d41f5d86f61f4ae99111fde71cace9dc
I20260618 23:35:19.673023 17311 raft_consensus.cc:2726] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 23:35:20.431929 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:20.431679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.1:44033)
0618 23:35:20.431824 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:20.431877 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260618 23:35:20.508716 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:20.508407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.2:60683)
0618 23:35:20.508598 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:20.508649 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":94,"threads_started":1}
I20260618 23:35:20.657167 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:20.656768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.3:55921)
0618 23:35:20.656995 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:20.657063 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":97,"threads_started":1}
I20260618 23:35:21.432843 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:21.432529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.1:44107)
0618 23:35:21.432716 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:21.432780 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":68,"threads_started":1}
I20260618 23:35:21.509588 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:21.509308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.2:49639)
0618 23:35:21.509488 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:21.509534 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":71,"threads_started":1}
I20260618 23:35:21.658317 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:21.657972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.3:38425)
0618 23:35:21.658192 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:21.658259 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":88,"threads_started":1}
I20260618 23:35:21.977916 17318 raft_consensus.cc:482] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:35:21.978041 17318 raft_consensus.cc:504] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:35:21.978344 17318 leader_election.cc:288] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d41f5d86f61f4ae99111fde71cace9dc (127.0.98.3:44033), 43363f29c3184a7d9f6f916bf560a182 (127.0.98.2:44357)
W20260618 23:35:21.979019 16916 leader_election.cc:341] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d41f5d86f61f4ae99111fde71cace9dc (127.0.98.3:44033): Not found: Tablet not found: 1890ceeeeaca4588b3769893f5ec639c
W20260618 23:35:21.979117 16917 leader_election.cc:341] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43363f29c3184a7d9f6f916bf560a182 (127.0.98.2:44357): Not found: Tablet not found: 1890ceeeeaca4588b3769893f5ec639c
I20260618 23:35:21.979246 16917 leader_election.cc:302] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa6e88a6a1404de09e8beaf85d5d4deb; no voters: 43363f29c3184a7d9f6f916bf560a182, d41f5d86f61f4ae99111fde71cace9dc
I20260618 23:35:21.979349 17318 raft_consensus.cc:2726] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 23:35:22.433885 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:22.433517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.1:33775)
0618 23:35:22.433740 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:22.433809 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":105,"threads_started":1}
I20260618 23:35:22.510654 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:22.510328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.2:53509)
0618 23:35:22.510531 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:22.510601 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":113,"threads_started":1}
I20260618 23:35:22.659389 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:22.659059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.3:35485)
0618 23:35:22.659259 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:22.659325 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":71,"threads_started":1}
I20260618 23:35:23.434940 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:23.434601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.1:53649)
0618 23:35:23.434824 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:23.434889 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":95,"threads_started":1}
I20260618 23:35:23.511945 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:23.511476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.2:58447)
0618 23:35:23.511779 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:23.511857 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":146,"threads_started":1}
I20260618 23:35:23.660480 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:23.660110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.3:48663)
0618 23:35:23.660361 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:23.660424 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":111,"threads_started":1}
I20260618 23:35:24.436154 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:24.435825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.1:58577)
0618 23:35:24.436041 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:24.436105 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":98,"threads_started":1}
I20260618 23:35:24.512984 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:24.512612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.2:39481)
0618 23:35:24.512797 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:24.512905 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":71,"threads_started":1}
I20260618 23:35:24.661538 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:24.661145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.3:40989)
0618 23:35:24.661405 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:24.661475 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":95,"threads_started":1}
I20260618 23:35:25.437211 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:25.436842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.1:45743)
0618 23:35:25.437067 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:25.437138 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":80,"threads_started":1}
I20260618 23:35:25.514070 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:25.513752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.2:59877)
0618 23:35:25.513937 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:25.514004 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":89,"threads_started":1}
I20260618 23:35:25.662701 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:25.662328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.3:46545)
0618 23:35:25.662576 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:25.662644 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":102,"threads_started":1}
I20260618 23:35:26.438469 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:26.438008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.1:57181)
0618 23:35:26.438245 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:26.438357 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":129,"threads_started":1}
I20260618 23:35:26.514845 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:26.514561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.2:52477)
0618 23:35:26.514723 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:26.514768 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":82,"threads_started":1}
I20260618 23:35:26.532718 17333 raft_consensus.cc:482] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:35:26.532869 17333 raft_consensus.cc:504] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:35:26.533095 17333 leader_election.cc:288] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d41f5d86f61f4ae99111fde71cace9dc (127.0.98.3:44033), 43363f29c3184a7d9f6f916bf560a182 (127.0.98.2:44357)
W20260618 23:35:26.533663 16916 leader_election.cc:341] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d41f5d86f61f4ae99111fde71cace9dc (127.0.98.3:44033): Not found: Tablet not found: 1890ceeeeaca4588b3769893f5ec639c
W20260618 23:35:26.533809 16917 leader_election.cc:341] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43363f29c3184a7d9f6f916bf560a182 (127.0.98.2:44357): Not found: Tablet not found: 1890ceeeeaca4588b3769893f5ec639c
I20260618 23:35:26.533911 16917 leader_election.cc:302] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa6e88a6a1404de09e8beaf85d5d4deb; no voters: 43363f29c3184a7d9f6f916bf560a182, d41f5d86f61f4ae99111fde71cace9dc
I20260618 23:35:26.534143 17333 raft_consensus.cc:2726] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 23:35:26.663901 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:26.663587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.3:50509)
0618 23:35:26.663761 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:26.663834 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":76,"threads_started":1}
I20260618 23:35:27.439738 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:27.439297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.1:35585)
0618 23:35:27.439519 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:27.439673 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":88,"threads_started":1}
I20260618 23:35:27.515911 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:27.515482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.2:53355)
0618 23:35:27.515750 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:27.515834 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":181,"threads_started":1}
I20260618 23:35:27.665055 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:27.664667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.3:59217)
0618 23:35:27.664916 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:27.665002 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":96,"threads_started":1}
I20260618 23:35:28.440958 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:28.440579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.1:50405)
0618 23:35:28.440799 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:28.440879 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":81,"threads_started":1}
I20260618 23:35:28.516947 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:28.516594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.2:40627)
0618 23:35:28.516808 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:28.516893 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":90,"threads_started":1}
I20260618 23:35:28.666184 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:28.665844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.3:35513)
0618 23:35:28.666067 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:28.666134 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":102,"threads_started":1}
I20260618 23:35:29.441921 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:29.441594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.1:36323)
0618 23:35:29.441794 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:29.441857 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":83,"threads_started":1}
I20260618 23:35:29.517978 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:29.517664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.2:53773)
0618 23:35:29.517854 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:29.517914 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":74,"threads_started":1}
I20260618 23:35:29.667209 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:29.666869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.3:50895)
0618 23:35:29.667092 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:29.667158 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":98,"threads_started":1}
I20260618 23:35:30.093210 17344 raft_consensus.cc:482] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:35:30.093372 17344 raft_consensus.cc:504] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d41f5d86f61f4ae99111fde71cace9dc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44033 } } peers { permanent_uuid: "43363f29c3184a7d9f6f916bf560a182" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44357 } } peers { permanent_uuid: "aa6e88a6a1404de09e8beaf85d5d4deb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38713 } }
I20260618 23:35:30.093606 17344 leader_election.cc:288] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d41f5d86f61f4ae99111fde71cace9dc (127.0.98.3:44033), 43363f29c3184a7d9f6f916bf560a182 (127.0.98.2:44357)
W20260618 23:35:30.094143 16916 leader_election.cc:341] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d41f5d86f61f4ae99111fde71cace9dc (127.0.98.3:44033): Not found: Tablet not found: 1890ceeeeaca4588b3769893f5ec639c
W20260618 23:35:30.094257 16917 leader_election.cc:341] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43363f29c3184a7d9f6f916bf560a182 (127.0.98.2:44357): Not found: Tablet not found: 1890ceeeeaca4588b3769893f5ec639c
I20260618 23:35:30.094303 16917 leader_election.cc:302] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa6e88a6a1404de09e8beaf85d5d4deb; no voters: 43363f29c3184a7d9f6f916bf560a182, d41f5d86f61f4ae99111fde71cace9dc
I20260618 23:35:30.094424 17344 raft_consensus.cc:2726] T 1890ceeeeaca4588b3769893f5ec639c P aa6e88a6a1404de09e8beaf85d5d4deb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 23:35:30.443058 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:30.442668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.1:33665)
0618 23:35:30.442937 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:30.443005 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":108,"threads_started":1}
I20260618 23:35:30.519028 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:30.518677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.2:42225)
0618 23:35:30.518908 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:30.518974 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":141,"threads_started":1}
I20260618 23:35:30.668368 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:30.668039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.3:41607)
0618 23:35:30.668233 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:30.668305 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":76,"threads_started":1}
I20260618 23:35:31.444449 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:31.444007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.1:47395)
0618 23:35:31.444234 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:31.444302 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":103,"threads_started":1}
I20260618 23:35:31.520115 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:31.519814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.2:49539)
0618 23:35:31.519985 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:31.520050 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
I20260618 23:35:31.663075 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 16864
I20260618 23:35:31.668426 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 17032
I20260618 23:35:31.669610 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:31.669320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38187 (local address 127.0.98.3:45229)
0618 23:35:31.669516 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:31.669553 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":62,"threads_started":1}
I20260618 23:35:31.674495 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 17165
2026-06-18T23:35:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58423 ms)
[----------] 30 tests from SecurityITest (242071 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:35:31 dist-test-slave-6224 krb5kdc[17353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:35:31 dist-test-slave-6224 krb5kdc[17353](info): set up 2 sockets
Jun 18 23:35:31 dist-test-slave-6224 krb5kdc[17353](info): commencing operation
krb5kdc: starting...
W20260618 23:35:32.727528 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:35:32 dist-test-slave-6224 krb5kdc[17353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:35:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:35:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:35:32.862537 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34685
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:32.949120 17366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:32.949373 17366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:32.949399 17366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:32.950795 17366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:35:32.950863 17366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:32.950879 17366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:32.950894 17366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:35:32.950908 17366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:35:32.952770 17366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:32.953101 17366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:32.953363 17366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:35:32.959949 17366 server_base.cc:1034] running on GCE node
W20260618 23:35:32.959964 17372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:32.959975 17374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:32.960230 17371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:32.960645 17366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:32.961046 17366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:32.962273 17366 hybrid_clock.cc:648] HybridClock initialized: now 1781825732962246 us; error 55 us; skew 500 ppm
Jun 18 23:35:32 dist-test-slave-6224 krb5kdc[17353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825732, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:32.965382 17366 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:35:32.965960 17366 webserver.cc:460] Webserver started at http://127.0.98.62:45821/ using document root <none> and password file <none>
I20260618 23:35:32.966229 17366 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:32.966303 17366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:32.966470 17366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:32.967656 17366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "087b99b40dde4dc8a1abe2f5c027a7e3"
format_stamp: "Formatted at 2026-06-18 23:35:32 on dist-test-slave-6224"
server_key: "d2b1bb894d065eccc793d039462bc237"
server_key_iv: "b175c424d5c9cd21d4809b7e74d42a80"
server_key_version: "encryptionkey@0"
I20260618 23:35:32.968083 17366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "087b99b40dde4dc8a1abe2f5c027a7e3"
format_stamp: "Formatted at 2026-06-18 23:35:32 on dist-test-slave-6224"
server_key: "d2b1bb894d065eccc793d039462bc237"
server_key_iv: "b175c424d5c9cd21d4809b7e74d42a80"
server_key_version: "encryptionkey@0"
I20260618 23:35:32.969672 17366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260618 23:35:32.970592 17381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:32.970861 17366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:35:32.970978 17366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "087b99b40dde4dc8a1abe2f5c027a7e3"
format_stamp: "Formatted at 2026-06-18 23:35:32 on dist-test-slave-6224"
server_key: "d2b1bb894d065eccc793d039462bc237"
server_key_iv: "b175c424d5c9cd21d4809b7e74d42a80"
server_key_version: "encryptionkey@0"
I20260618 23:35:32.971071 17366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:32.985052 17366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:32.987627 17366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:32.987841 17366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:32.992635 17366 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34685
I20260618 23:35:32.992708 17433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34685 every 8 connection(s)
I20260618 23:35:32.993134 17366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:35:32.994375 17434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:32.998359 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 17366
I20260618 23:35:32.998374 17434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3: Bootstrap starting.
I20260618 23:35:32.998505 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:35:32.998749 392 external_mini_cluster.cc:1428] Setting key f89b91a3672c74e6edb9fa136c01e81d
I20260618 23:35:32.999284 17434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:32.999747 17434 log.cc:824] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:33.000690 17434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3: No bootstrap required, opened a new log
I20260618 23:35:33.003262 17434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087b99b40dde4dc8a1abe2f5c027a7e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34685 } }
I20260618 23:35:33.003445 17434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:33.003470 17434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 087b99b40dde4dc8a1abe2f5c027a7e3, State: Initialized, Role: FOLLOWER
I20260618 23:35:33.003649 17434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087b99b40dde4dc8a1abe2f5c027a7e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34685 } }
I20260618 23:35:33.003743 17434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:35:33.003777 17434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:35:33.003831 17434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:33.004653 17434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087b99b40dde4dc8a1abe2f5c027a7e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34685 } }
I20260618 23:35:33.004782 17434 leader_election.cc:302] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 087b99b40dde4dc8a1abe2f5c027a7e3; no voters:
I20260618 23:35:33.004987 17434 leader_election.cc:288] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:35:33.005107 17439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:33.005414 17434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:35:33.005426 17439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3 [term 1 LEADER]: Becoming Leader. State: Replica: 087b99b40dde4dc8a1abe2f5c027a7e3, State: Running, Role: LEADER
I20260618 23:35:33.005759 17439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087b99b40dde4dc8a1abe2f5c027a7e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34685 } }
I20260618 23:35:33.006326 17441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "087b99b40dde4dc8a1abe2f5c027a7e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087b99b40dde4dc8a1abe2f5c027a7e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34685 } } }
I20260618 23:35:33.006422 17441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:33.006590 17441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 087b99b40dde4dc8a1abe2f5c027a7e3. Latest consensus state: current_term: 1 leader_uuid: "087b99b40dde4dc8a1abe2f5c027a7e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "087b99b40dde4dc8a1abe2f5c027a7e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34685 } } }
I20260618 23:35:33.006654 17441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:33.006748 17448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:35:33.007354 17448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 18 23:35:33 dist-test-slave-6224 krb5kdc[17353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:35:33.009213 17448 catalog_manager.cc:1347] Generated new cluster ID: b25ca019813142b28bd5af2477075611
I20260618 23:35:33.009276 17448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:35:33.013012 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:32.999420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33454 (local address 127.0.98.62:34685)
0618 23:35:32.999733 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:32.999744 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:32.999785 (+ 41us) server_negotiation.cc:407] Connection header received
0618 23:35:33.000507 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:33.000517 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:33.000591 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:33.000794 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:33.001516 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.003112 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.004255 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.004962 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.005529 (+ 567us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:33.008818 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:33.008825 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:33.008826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:33.008844 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:33.011035 (+ 2191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:33.011608 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:33.011612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:33.011612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:33.011730 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:33.011998 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:33.011998 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:33.011999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:33.012316 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:33.012497 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:33.012636 (+ 139us) server_negotiation.cc:299] Negotiation successful
0618 23:35:33.012787 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":155,"threads_started":1}
I20260618 23:35:33.022472 17448 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:35:33.023082 17448 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:35:33.029703 17448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3: Generated new TSK 0
I20260618 23:35:33.029907 17448 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260618 23:35:33.061033 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34685
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:33.145347 17460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:33.145648 17460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:33.145689 17460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:33.146945 17460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:33.147037 17460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:33.147084 17460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:35:33.148718 17460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:33.149044 17460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:33.149296 17460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:33.155153 17466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:33.155373 17468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:33.155186 17460 server_base.cc:1034] running on GCE node
W20260618 23:35:33.155193 17465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:33.155989 17460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:33.156402 17460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:33.157619 17460 hybrid_clock.cc:648] HybridClock initialized: now 1781825733157564 us; error 101 us; skew 500 ppm
Jun 18 23:35:33 dist-test-slave-6224 krb5kdc[17353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:33.161337 17460 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:35:33.161990 17460 webserver.cc:460] Webserver started at http://127.0.98.1:45375/ using document root <none> and password file <none>
I20260618 23:35:33.162297 17460 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:33.162379 17460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:33.162549 17460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:33.164002 17460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "277339c7093f4d73a767d2102c851568"
format_stamp: "Formatted at 2026-06-18 23:35:33 on dist-test-slave-6224"
server_key: "72303aa19da444a8a14d051c43d01bb5"
server_key_iv: "2f1def9982c5d997744bcd3a621a9fb5"
server_key_version: "encryptionkey@0"
I20260618 23:35:33.164451 17460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "277339c7093f4d73a767d2102c851568"
format_stamp: "Formatted at 2026-06-18 23:35:33 on dist-test-slave-6224"
server_key: "72303aa19da444a8a14d051c43d01bb5"
server_key_iv: "2f1def9982c5d997744bcd3a621a9fb5"
server_key_version: "encryptionkey@0"
I20260618 23:35:33.166244 17460 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260618 23:35:33.167484 17475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:33.167857 17460 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:33.167985 17460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "277339c7093f4d73a767d2102c851568"
format_stamp: "Formatted at 2026-06-18 23:35:33 on dist-test-slave-6224"
server_key: "72303aa19da444a8a14d051c43d01bb5"
server_key_iv: "2f1def9982c5d997744bcd3a621a9fb5"
server_key_version: "encryptionkey@0"
I20260618 23:35:33.168081 17460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:33.184103 17460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:33.186664 17460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:33.186981 17460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:33.187300 17460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:35:33.187808 17460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:35:33.187867 17460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:33.187913 17460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:35:33.187947 17460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:33.196094 17460 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45557
I20260618 23:35:33.196285 17588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45557 every 8 connection(s)
I20260618 23:35:33.196635 17460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:35:33.197927 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 17460
I20260618 23:35:33.198071 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:35:33.198316 392 external_mini_cluster.cc:1428] Setting key 581a108bb78e6e828b672f3669fa319f
Jun 18 23:35:33 dist-test-slave-6224 krb5kdc[17353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260618 23:35:33.208602 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:33.197715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48101 (local address 127.0.98.62:34685)
0618 23:35:33.197803 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:33.197805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:33.198178 (+ 373us) server_negotiation.cc:407] Connection header received
0618 23:35:33.199567 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:33.199568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:33.199581 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:33.199631 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:33.200441 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.200992 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.202082 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.202301 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.202405 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:33.204840 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:33.204842 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:33.204842 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:33.204850 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:33.206956 (+ 2106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:33.207444 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:33.207446 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:33.207446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:33.207506 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:33.207909 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:33.207910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:33.207910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:33.208037 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:33.208115 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:33.208398 (+ 283us) server_negotiation.cc:299] Negotiation successful
0618 23:35:33.208494 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 23:35:33.208935 17591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:33.197633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34685 (local address 127.0.98.1:48101)
0618 23:35:33.197932 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:33.197981 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:33.199432 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:33.199716 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:33.199720 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:33.200072 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:35:33.200324 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:33.200329 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.201105 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.201105 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:33.201980 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:33.201982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.202407 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.202408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:33.202550 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:33.203173 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:35:33.203182 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:35:33.204723 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:35:33.207066 (+ 2343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:35:33.207075 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:35:33.207079 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:35:33.207348 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:35:33.207691 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:35:33.207692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:35:33.207692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:35:33.207820 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:35:33.208107 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:35:33.208113 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:35:33.208261 (+ 148us) client_negotiation.cc:769] Sending connection context
0618 23:35:33.208377 (+ 116us) client_negotiation.cc:240] Negotiation successful
0618 23:35:33.208608 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":141,"threads_started":1}
I20260618 23:35:33.209750 17589 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34685
I20260618 23:35:33.209864 17589 heartbeater.cc:464] Registering TS with master...
I20260618 23:35:33.210037 17589 heartbeater.cc:511] Master 127.0.98.62:34685 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260618 23:35:33.210793 17398 ts_manager.cc:194] Registered new tserver with Master: 277339c7093f4d73a767d2102c851568 (127.0.98.1:45557)
I20260618 23:35:33.211628 17398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260618 23:35:33.228044 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34685
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:33.323007 17594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:33.323496 17594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:33.323552 17594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:33.325275 17594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:33.325388 17594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:33.325451 17594 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:35:33.327986 17594 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17594
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:33.328380 17594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:33.328694 17594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:33.335417 17599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:33.335423 17600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:33.335870 17594 server_base.cc:1034] running on GCE node
W20260618 23:35:33.335423 17602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:33.336386 17594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:33.336743 17594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:33.337965 17594 hybrid_clock.cc:648] HybridClock initialized: now 1781825733337925 us; error 53 us; skew 500 ppm
Jun 18 23:35:33 dist-test-slave-6224 krb5kdc[17353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:33.341445 17594 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:35:33.342087 17594 webserver.cc:460] Webserver started at http://127.0.98.2:32915/ using document root <none> and password file <none>
I20260618 23:35:33.342418 17594 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:33.342505 17594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:33.342674 17594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:33.344062 17594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "4cf154ba57c64e0c93ec1ff053467f2e"
format_stamp: "Formatted at 2026-06-18 23:35:33 on dist-test-slave-6224"
server_key: "d0b9278acfc16a7938edfa9f054b0f87"
server_key_iv: "8abc591deff34f006914cb25d17c70bb"
server_key_version: "encryptionkey@0"
I20260618 23:35:33.344470 17594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4cf154ba57c64e0c93ec1ff053467f2e"
format_stamp: "Formatted at 2026-06-18 23:35:33 on dist-test-slave-6224"
server_key: "d0b9278acfc16a7938edfa9f054b0f87"
server_key_iv: "8abc591deff34f006914cb25d17c70bb"
server_key_version: "encryptionkey@0"
I20260618 23:35:33.346117 17594 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:35:33.347069 17609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:33.347440 17594 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:33.347599 17594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "4cf154ba57c64e0c93ec1ff053467f2e"
format_stamp: "Formatted at 2026-06-18 23:35:33 on dist-test-slave-6224"
server_key: "d0b9278acfc16a7938edfa9f054b0f87"
server_key_iv: "8abc591deff34f006914cb25d17c70bb"
server_key_version: "encryptionkey@0"
I20260618 23:35:33.347699 17594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:33.379259 17594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:33.382246 17594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:33.382439 17594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:33.382745 17594 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:35:33.383129 17594 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:35:33.383193 17594 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:33.383239 17594 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:35:33.383263 17594 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:33.391512 17594 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44653
I20260618 23:35:33.391582 17722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44653 every 8 connection(s)
I20260618 23:35:33.392087 17594 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:35:33.394915 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 17594
I20260618 23:35:33.395026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:35:33.395316 392 external_mini_cluster.cc:1428] Setting key fa930da0e5eb405312c7d0b52f6125ad
Jun 18 23:35:33 dist-test-slave-6224 krb5kdc[17353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:35:33.405651 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:33.393109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42417 (local address 127.0.98.62:34685)
0618 23:35:33.393175 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:33.393177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:33.393396 (+ 219us) server_negotiation.cc:407] Connection header received
0618 23:35:33.394509 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:33.394510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:33.394525 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:33.394595 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:33.395508 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.396167 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.398086 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.398379 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.398501 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:33.401600 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:33.401602 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:33.401602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:33.401611 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:33.403407 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:33.404227 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:33.404228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:33.404229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:33.404300 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:33.404722 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:33.404724 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:33.404724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:33.404864 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:33.404941 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:33.405421 (+ 480us) server_negotiation.cc:299] Negotiation successful
0618 23:35:33.405522 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 23:35:33.405947 17725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:33.392980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34685 (local address 127.0.98.2:42417)
0618 23:35:33.393200 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:33.393258 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:33.394372 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:33.394684 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:33.394688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:33.395073 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:35:33.395327 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:33.395332 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.396325 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.396327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:33.397919 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:33.397925 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.398534 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.398537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:33.398727 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:33.399394 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:35:33.399403 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:35:33.401432 (+ 2029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:35:33.403588 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:35:33.403604 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:35:33.403609 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:35:33.404075 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:35:33.404401 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:35:33.404403 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:35:33.404403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:35:33.404592 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:35:33.404963 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:35:33.404971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:35:33.405230 (+ 259us) client_negotiation.cc:769] Sending connection context
0618 23:35:33.405380 (+ 150us) client_negotiation.cc:240] Negotiation successful
0618 23:35:33.405634 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260618 23:35:33.406848 17723 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34685
I20260618 23:35:33.406984 17723 heartbeater.cc:464] Registering TS with master...
I20260618 23:35:33.407239 17723 heartbeater.cc:511] Master 127.0.98.62:34685 requested a full tablet report, sending...
I20260618 23:35:33.407874 17398 ts_manager.cc:194] Registered new tserver with Master: 4cf154ba57c64e0c93ec1ff053467f2e (127.0.98.2:44653)
I20260618 23:35:33.408393 17398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260618 23:35:33.428020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34685
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:33.521106 17728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:33.521401 17728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:33.521447 17728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:33.522720 17728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:33.522814 17728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:33.522872 17728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:35:33.524689 17728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34685
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:33.525025 17728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:33.525288 17728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:33.531742 17733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:33.532171 17736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:33.532208 17734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:33.532215 17728 server_base.cc:1034] running on GCE node
I20260618 23:35:33.532550 17728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:33.532902 17728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:33.534142 17728 hybrid_clock.cc:648] HybridClock initialized: now 1781825733534114 us; error 46 us; skew 500 ppm
Jun 18 23:35:33 dist-test-slave-6224 krb5kdc[17353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:33.537633 17728 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:35:33.538223 17728 webserver.cc:460] Webserver started at http://127.0.98.3:45553/ using document root <none> and password file <none>
I20260618 23:35:33.538486 17728 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:33.538538 17728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:33.538694 17728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:33.539968 17728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "d4b02bb2d1b941fea8965627ccd0e587"
format_stamp: "Formatted at 2026-06-18 23:35:33 on dist-test-slave-6224"
server_key: "c5e3068ba1a17ba31a116cf6c5782c95"
server_key_iv: "256318cb76cf2bc234b31465cc6b60f3"
server_key_version: "encryptionkey@0"
I20260618 23:35:33.540516 17728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d4b02bb2d1b941fea8965627ccd0e587"
format_stamp: "Formatted at 2026-06-18 23:35:33 on dist-test-slave-6224"
server_key: "c5e3068ba1a17ba31a116cf6c5782c95"
server_key_iv: "256318cb76cf2bc234b31465cc6b60f3"
server_key_version: "encryptionkey@0"
I20260618 23:35:33.542807 17728 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260618 23:35:33.543984 17743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:33.544286 17728 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:33.544420 17728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "d4b02bb2d1b941fea8965627ccd0e587"
format_stamp: "Formatted at 2026-06-18 23:35:33 on dist-test-slave-6224"
server_key: "c5e3068ba1a17ba31a116cf6c5782c95"
server_key_iv: "256318cb76cf2bc234b31465cc6b60f3"
server_key_version: "encryptionkey@0"
I20260618 23:35:33.544519 17728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:33.560988 17728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:33.563771 17728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:33.563975 17728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:33.564265 17728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:35:33.564762 17728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:35:33.564831 17728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:33.564942 17728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:35:33.564973 17728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:33.572754 17728 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44875
I20260618 23:35:33.572913 17856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44875 every 8 connection(s)
I20260618 23:35:33.573248 17728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:35:33.575289 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 17728
I20260618 23:35:33.575400 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:35:33.575672 392 external_mini_cluster.cc:1428] Setting key efc92ca18b8b5189303b46dcef5206bf
Jun 18 23:35:33 dist-test-slave-6224 krb5kdc[17353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825733, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:35:33.585807 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:33.574332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56771 (local address 127.0.98.62:34685)
0618 23:35:33.574397 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:33.574399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:33.574709 (+ 310us) server_negotiation.cc:407] Connection header received
0618 23:35:33.576013 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:33.576014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:33.576029 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:33.576130 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:33.577127 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.577782 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.578923 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.579192 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.579270 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:33.582265 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:33.582266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:33.582267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:33.582276 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:33.583983 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:33.584542 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:33.584544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:33.584544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:33.584612 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:33.584978 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:33.584979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:33.584979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:33.585153 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:33.585251 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:33.585616 (+ 365us) server_negotiation.cc:299] Negotiation successful
0618 23:35:33.585722 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260618 23:35:33.585973 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:33.574344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34685 (local address 127.0.98.3:56771)
0618 23:35:33.574559 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:33.574605 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:33.575864 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:33.576254 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:33.576259 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:33.576707 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:35:33.577038 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:33.577045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.577904 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.577904 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:33.578811 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:33.578814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.579325 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.579326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:33.579466 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:33.580207 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:35:33.580217 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:35:33.582133 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:35:33.584112 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:35:33.584122 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:35:33.584124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:35:33.584444 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:35:33.584714 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:35:33.584715 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:35:33.584715 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:35:33.584847 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:35:33.585279 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:35:33.585285 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:35:33.585444 (+ 159us) client_negotiation.cc:769] Sending connection context
0618 23:35:33.585564 (+ 120us) client_negotiation.cc:240] Negotiation successful
0618 23:35:33.585715 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":136,"threads_started":1}
I20260618 23:35:33.586850 17857 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34685
I20260618 23:35:33.586979 17857 heartbeater.cc:464] Registering TS with master...
I20260618 23:35:33.587278 17857 heartbeater.cc:511] Master 127.0.98.62:34685 requested a full tablet report, sending...
I20260618 23:35:33.587873 17398 ts_manager.cc:194] Registered new tserver with Master: d4b02bb2d1b941fea8965627ccd0e587 (127.0.98.3:44875)
I20260618 23:35:33.588465 17398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56771
I20260618 23:35:33.588471 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:35:33.598230 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:33.590060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33466 (local address 127.0.98.62:34685)
0618 23:35:33.590135 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:33.590137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:33.590221 (+ 84us) server_negotiation.cc:407] Connection header received
0618 23:35:33.590291 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:33.590292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:33.590304 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:33.590361 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:33.591121 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.591725 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.592581 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.592784 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.592882 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:33.594545 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:33.594547 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:33.594547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:33.594556 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:33.596709 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:33.597254 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:33.597256 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:33.597256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:33.597344 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:33.597593 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:33.597594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:33.597595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:33.597798 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:33.598015 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:33.598060 (+ 45us) server_negotiation.cc:299] Negotiation successful
0618 23:35:33.598152 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 23:35:33.601135 17398 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:35:33.603161 17398 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:35:33.612172 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:33.606912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44653 (local address 127.0.0.1:55970)
0618 23:35:33.607311 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:33.607322 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:33.607386 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:33.608254 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:33.608255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:33.608257 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:33.608312 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:33.608314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.609650 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.609651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:33.610716 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:33.610719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.611750 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.611751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:33.611850 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:33.611870 (+ 20us) client_negotiation.cc:769] Sending connection context
0618 23:35:33.611972 (+ 102us) client_negotiation.cc:240] Negotiation successful
0618 23:35:33.612062 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":3,"thread_start_us":136,"threads_started":1}
I20260618 23:35:33.612294 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:33.606880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44875 (local address 127.0.0.1:50460)
0618 23:35:33.607235 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:33.607295 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:33.607433 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:33.607745 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:33.607749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:33.607752 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:33.607867 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:33.607870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.609171 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.609172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:33.610881 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:33.610885 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.611964 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.611965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:33.612048 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:33.612054 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:35:33.612127 (+ 73us) client_negotiation.cc:240] Negotiation successful
0618 23:35:33.612208 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":4,"thread_start_us":95,"threads_started":1}
I20260618 23:35:33.612639 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:33.607016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50460 (local address 127.0.98.3:44875)
0618 23:35:33.607217 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:33.607225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:33.607404 (+ 179us) server_negotiation.cc:407] Connection header received
0618 23:35:33.607537 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:33.607540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:33.607606 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:33.607695 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:33.608008 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.609037 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.611000 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.611778 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.611850 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:33.611902 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:33.612484 (+ 582us) server_negotiation.cc:299] Negotiation successful
0618 23:35:33.612541 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":97,"threads_started":1}
I20260618 23:35:33.612169 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:33.607050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55970 (local address 127.0.98.2:44653)
0618 23:35:33.607952 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:33.607958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:33.607970 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:35:33.608018 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:33.608021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:33.608039 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:33.608142 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:33.608437 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.609540 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.610813 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.611640 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.611722 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:33.611795 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:33.612001 (+ 206us) server_negotiation.cc:299] Negotiation successful
0618 23:35:33.612051 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":119,"threads_started":1}
I20260618 23:35:33.613025 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:33.607167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45557 (local address 127.0.0.1:40628)
0618 23:35:33.607362 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:33.607378 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:33.607458 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:33.608034 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:33.608036 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:33.608038 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:33.608120 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:33.608123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.609479 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.609480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:33.610883 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:33.610885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.612746 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.612747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:33.612819 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:33.612824 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:35:33.612891 (+ 67us) client_negotiation.cc:240] Negotiation successful
0618 23:35:33.612947 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":98,"thread_start_us":101,"threads_started":1}
I20260618 23:35:33.613590 17791 tablet_service.cc:1467] Processing CreateTablet for tablet 6b6f5edfafc842b9be921ece2ffc7573 (DEFAULT_TABLE table=test-table [id=df32460d111549a896063bd216081629]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:35:33.613682 17657 tablet_service.cc:1467] Processing CreateTablet for tablet 6b6f5edfafc842b9be921ece2ffc7573 (DEFAULT_TABLE table=test-table [id=df32460d111549a896063bd216081629]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:35:33.614078 17791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b6f5edfafc842b9be921ece2ffc7573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:33.614078 17657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b6f5edfafc842b9be921ece2ffc7573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:33.618757 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:33.607471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40628 (local address 127.0.98.1:45557)
0618 23:35:33.607741 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:33.607746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:33.607758 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:35:33.607798 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:33.607800 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:33.607814 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:33.607925 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:33.608181 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.609326 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.611717 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.612565 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.612645 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:33.612708 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:33.612912 (+ 204us) server_negotiation.cc:299] Negotiation successful
0618 23:35:33.618636 (+ 5724us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":146,"threads_started":1}
I20260618 23:35:33.618752 17874 tablet_bootstrap.cc:492] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587: Bootstrap starting.
I20260618 23:35:33.619716 17523 tablet_service.cc:1467] Processing CreateTablet for tablet 6b6f5edfafc842b9be921ece2ffc7573 (DEFAULT_TABLE table=test-table [id=df32460d111549a896063bd216081629]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:35:33.620188 17523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b6f5edfafc842b9be921ece2ffc7573. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:33.620527 17874 tablet_bootstrap.cc:654] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:33.621209 17874 log.cc:824] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:33.621711 17875 tablet_bootstrap.cc:492] T 6b6f5edfafc842b9be921ece2ffc7573 P 4cf154ba57c64e0c93ec1ff053467f2e: Bootstrap starting.
I20260618 23:35:33.622579 17875 tablet_bootstrap.cc:654] T 6b6f5edfafc842b9be921ece2ffc7573 P 4cf154ba57c64e0c93ec1ff053467f2e: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:33.623246 17875 log.cc:824] T 6b6f5edfafc842b9be921ece2ffc7573 P 4cf154ba57c64e0c93ec1ff053467f2e: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:33.624017 17874 tablet_bootstrap.cc:492] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587: No bootstrap required, opened a new log
I20260618 23:35:33.624135 17874 ts_tablet_manager.cc:1397] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260618 23:35:33.624522 17875 tablet_bootstrap.cc:492] T 6b6f5edfafc842b9be921ece2ffc7573 P 4cf154ba57c64e0c93ec1ff053467f2e: No bootstrap required, opened a new log
I20260618 23:35:33.624688 17875 ts_tablet_manager.cc:1397] T 6b6f5edfafc842b9be921ece2ffc7573 P 4cf154ba57c64e0c93ec1ff053467f2e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 23:35:33.625289 17878 tablet_bootstrap.cc:492] T 6b6f5edfafc842b9be921ece2ffc7573 P 277339c7093f4d73a767d2102c851568: Bootstrap starting.
I20260618 23:35:33.626616 17878 tablet_bootstrap.cc:654] T 6b6f5edfafc842b9be921ece2ffc7573 P 277339c7093f4d73a767d2102c851568: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:33.627236 17878 log.cc:824] T 6b6f5edfafc842b9be921ece2ffc7573 P 277339c7093f4d73a767d2102c851568: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:33.627112 17874 raft_consensus.cc:348] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b02bb2d1b941fea8965627ccd0e587" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44875 } } peers { permanent_uuid: "4cf154ba57c64e0c93ec1ff053467f2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "277339c7093f4d73a767d2102c851568" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45557 } }
I20260618 23:35:33.627386 17874 raft_consensus.cc:374] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:33.627437 17874 raft_consensus.cc:729] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4b02bb2d1b941fea8965627ccd0e587, State: Initialized, Role: FOLLOWER
I20260618 23:35:33.627633 17874 consensus_queue.cc:260] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b02bb2d1b941fea8965627ccd0e587" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44875 } } peers { permanent_uuid: "4cf154ba57c64e0c93ec1ff053467f2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "277339c7093f4d73a767d2102c851568" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45557 } }
I20260618 23:35:33.627722 17875 raft_consensus.cc:348] T 6b6f5edfafc842b9be921ece2ffc7573 P 4cf154ba57c64e0c93ec1ff053467f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b02bb2d1b941fea8965627ccd0e587" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44875 } } peers { permanent_uuid: "4cf154ba57c64e0c93ec1ff053467f2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "277339c7093f4d73a767d2102c851568" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45557 } }
I20260618 23:35:33.627998 17875 raft_consensus.cc:374] T 6b6f5edfafc842b9be921ece2ffc7573 P 4cf154ba57c64e0c93ec1ff053467f2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:33.628046 17874 ts_tablet_manager.cc:1428] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260618 23:35:33.628050 17875 raft_consensus.cc:729] T 6b6f5edfafc842b9be921ece2ffc7573 P 4cf154ba57c64e0c93ec1ff053467f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cf154ba57c64e0c93ec1ff053467f2e, State: Initialized, Role: FOLLOWER
I20260618 23:35:33.628139 17857 heartbeater.cc:503] Master 127.0.98.62:34685 was elected leader, sending a full tablet report...
I20260618 23:35:33.628201 17875 consensus_queue.cc:260] T 6b6f5edfafc842b9be921ece2ffc7573 P 4cf154ba57c64e0c93ec1ff053467f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b02bb2d1b941fea8965627ccd0e587" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44875 } } peers { permanent_uuid: "4cf154ba57c64e0c93ec1ff053467f2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "277339c7093f4d73a767d2102c851568" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45557 } }
I20260618 23:35:33.628525 17875 ts_tablet_manager.cc:1428] T 6b6f5edfafc842b9be921ece2ffc7573 P 4cf154ba57c64e0c93ec1ff053467f2e: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260618 23:35:33.628588 17723 heartbeater.cc:503] Master 127.0.98.62:34685 was elected leader, sending a full tablet report...
I20260618 23:35:33.629348 17878 tablet_bootstrap.cc:492] T 6b6f5edfafc842b9be921ece2ffc7573 P 277339c7093f4d73a767d2102c851568: No bootstrap required, opened a new log
I20260618 23:35:33.629451 17878 ts_tablet_manager.cc:1397] T 6b6f5edfafc842b9be921ece2ffc7573 P 277339c7093f4d73a767d2102c851568: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260618 23:35:33.631873 17878 raft_consensus.cc:348] T 6b6f5edfafc842b9be921ece2ffc7573 P 277339c7093f4d73a767d2102c851568 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b02bb2d1b941fea8965627ccd0e587" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44875 } } peers { permanent_uuid: "4cf154ba57c64e0c93ec1ff053467f2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "277339c7093f4d73a767d2102c851568" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45557 } }
I20260618 23:35:33.632040 17878 raft_consensus.cc:374] T 6b6f5edfafc842b9be921ece2ffc7573 P 277339c7093f4d73a767d2102c851568 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:33.632072 17878 raft_consensus.cc:729] T 6b6f5edfafc842b9be921ece2ffc7573 P 277339c7093f4d73a767d2102c851568 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 277339c7093f4d73a767d2102c851568, State: Initialized, Role: FOLLOWER
I20260618 23:35:33.632215 17878 consensus_queue.cc:260] T 6b6f5edfafc842b9be921ece2ffc7573 P 277339c7093f4d73a767d2102c851568 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b02bb2d1b941fea8965627ccd0e587" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44875 } } peers { permanent_uuid: "4cf154ba57c64e0c93ec1ff053467f2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "277339c7093f4d73a767d2102c851568" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45557 } }
I20260618 23:35:33.632545 17878 ts_tablet_manager.cc:1428] T 6b6f5edfafc842b9be921ece2ffc7573 P 277339c7093f4d73a767d2102c851568: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:35:33.632705 17589 heartbeater.cc:503] Master 127.0.98.62:34685 was elected leader, sending a full tablet report...
W20260618 23:35:33.642812 17724 tablet.cc:2369] T 6b6f5edfafc842b9be921ece2ffc7573 P 4cf154ba57c64e0c93ec1ff053467f2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:35:33.689291 17880 raft_consensus.cc:482] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:35:33.689514 17880 raft_consensus.cc:504] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b02bb2d1b941fea8965627ccd0e587" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44875 } } peers { permanent_uuid: "4cf154ba57c64e0c93ec1ff053467f2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "277339c7093f4d73a767d2102c851568" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45557 } }
I20260618 23:35:33.690143 17880 leader_election.cc:288] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4cf154ba57c64e0c93ec1ff053467f2e (127.0.98.2:44653), 277339c7093f4d73a767d2102c851568 (127.0.98.1:45557)
I20260618 23:35:33.693755 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:33.690384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44653 (local address 127.0.98.3:54193)
0618 23:35:33.690459 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:33.690476 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:33.690542 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:33.690776 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:33.690778 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:33.690781 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:33.690887 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:33.690890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.691686 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.691687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:33.692712 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:33.692715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.693368 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.693370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:33.693479 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:33.693486 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 23:35:33.693575 (+ 89us) client_negotiation.cc:240] Negotiation successful
0618 23:35:33.693641 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":32}
I20260618 23:35:33.693743 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:33.690475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54193 (local address 127.0.98.2:44653)
0618 23:35:33.690573 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:33.690576 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:33.690596 (+ 20us) server_negotiation.cc:407] Connection header received
0618 23:35:33.690640 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:33.690641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:33.690656 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:33.690742 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:33.690982 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.691593 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.692828 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.693253 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.693317 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:33.693336 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:33.693599 (+ 263us) server_negotiation.cc:299] Negotiation successful
0618 23:35:33.693659 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260618 23:35:33.694075 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:33.690620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58145 (local address 127.0.98.1:45557)
0618 23:35:33.690684 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:33.690686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:33.690755 (+ 69us) server_negotiation.cc:407] Connection header received
0618 23:35:33.690837 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:33.690838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:33.690856 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:33.690927 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:33.691161 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.691836 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.692996 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.693581 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.693666 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:33.693680 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:33.693916 (+ 236us) server_negotiation.cc:299] Negotiation successful
0618 23:35:33.694000 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 23:35:33.694332 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:33.690477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45557 (local address 127.0.98.3:58145)
0618 23:35:33.690684 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:33.690696 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:33.690766 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:33.690973 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:33.690974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:33.690977 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:33.691066 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:33.691069 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.691938 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.691939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:33.692847 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:33.692850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.693690 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.693691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:33.693776 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:33.693782 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:35:33.694187 (+ 405us) client_negotiation.cc:240] Negotiation successful
0618 23:35:33.694240 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":118,"threads_started":1}
I20260618 23:35:33.694263 17677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6f5edfafc842b9be921ece2ffc7573" candidate_uuid: "d4b02bb2d1b941fea8965627ccd0e587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf154ba57c64e0c93ec1ff053467f2e" is_pre_election: true
I20260618 23:35:33.694468 17677 raft_consensus.cc:2454] T 6b6f5edfafc842b9be921ece2ffc7573 P 4cf154ba57c64e0c93ec1ff053467f2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4b02bb2d1b941fea8965627ccd0e587 in term 0.
I20260618 23:35:33.694811 17744 leader_election.cc:302] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cf154ba57c64e0c93ec1ff053467f2e, d4b02bb2d1b941fea8965627ccd0e587; no voters:
I20260618 23:35:33.694886 17543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6f5edfafc842b9be921ece2ffc7573" candidate_uuid: "d4b02bb2d1b941fea8965627ccd0e587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "277339c7093f4d73a767d2102c851568" is_pre_election: true
I20260618 23:35:33.695050 17543 raft_consensus.cc:2454] T 6b6f5edfafc842b9be921ece2ffc7573 P 277339c7093f4d73a767d2102c851568 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4b02bb2d1b941fea8965627ccd0e587 in term 0.
I20260618 23:35:33.695295 17880 raft_consensus.cc:2781] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:35:33.695382 17880 raft_consensus.cc:482] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:35:33.695410 17880 raft_consensus.cc:3037] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:33.696656 17880 raft_consensus.cc:504] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b02bb2d1b941fea8965627ccd0e587" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44875 } } peers { permanent_uuid: "4cf154ba57c64e0c93ec1ff053467f2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "277339c7093f4d73a767d2102c851568" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45557 } }
I20260618 23:35:33.696883 17880 leader_election.cc:288] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587 [CANDIDATE]: Term 1 election: Requested vote from peers 4cf154ba57c64e0c93ec1ff053467f2e (127.0.98.2:44653), 277339c7093f4d73a767d2102c851568 (127.0.98.1:45557)
I20260618 23:35:33.697089 17677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6f5edfafc842b9be921ece2ffc7573" candidate_uuid: "d4b02bb2d1b941fea8965627ccd0e587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cf154ba57c64e0c93ec1ff053467f2e"
I20260618 23:35:33.697184 17677 raft_consensus.cc:3037] T 6b6f5edfafc842b9be921ece2ffc7573 P 4cf154ba57c64e0c93ec1ff053467f2e [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:33.697208 17543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b6f5edfafc842b9be921ece2ffc7573" candidate_uuid: "d4b02bb2d1b941fea8965627ccd0e587" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "277339c7093f4d73a767d2102c851568"
I20260618 23:35:33.697309 17543 raft_consensus.cc:3037] T 6b6f5edfafc842b9be921ece2ffc7573 P 277339c7093f4d73a767d2102c851568 [term 0 FOLLOWER]: Advancing to term 1
W20260618 23:35:33.697674 17590 tablet.cc:2369] T 6b6f5edfafc842b9be921ece2ffc7573 P 277339c7093f4d73a767d2102c851568: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:35:33.698256 17677 raft_consensus.cc:2454] T 6b6f5edfafc842b9be921ece2ffc7573 P 4cf154ba57c64e0c93ec1ff053467f2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4b02bb2d1b941fea8965627ccd0e587 in term 1.
I20260618 23:35:33.698302 17543 raft_consensus.cc:2454] T 6b6f5edfafc842b9be921ece2ffc7573 P 277339c7093f4d73a767d2102c851568 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4b02bb2d1b941fea8965627ccd0e587 in term 1.
I20260618 23:35:33.698503 17747 leader_election.cc:302] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 277339c7093f4d73a767d2102c851568, d4b02bb2d1b941fea8965627ccd0e587; no voters:
I20260618 23:35:33.698598 17880 raft_consensus.cc:2781] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:33.698750 17880 raft_consensus.cc:686] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587 [term 1 LEADER]: Becoming Leader. State: Replica: d4b02bb2d1b941fea8965627ccd0e587, State: Running, Role: LEADER
I20260618 23:35:33.698872 17880 consensus_queue.cc:237] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b02bb2d1b941fea8965627ccd0e587" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44875 } } peers { permanent_uuid: "4cf154ba57c64e0c93ec1ff053467f2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } } peers { permanent_uuid: "277339c7093f4d73a767d2102c851568" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45557 } }
I20260618 23:35:33.700067 17398 catalog_manager.cc:5495] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4b02bb2d1b941fea8965627ccd0e587 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "d4b02bb2d1b941fea8965627ccd0e587" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4b02bb2d1b941fea8965627ccd0e587" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cf154ba57c64e0c93ec1ff053467f2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "277339c7093f4d73a767d2102c851568" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45557 } health_report { overall_health: UNKNOWN } } }
I20260618 23:35:33.710143 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:33.706959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50476 (local address 127.0.98.3:44875)
0618 23:35:33.707283 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:33.707286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:33.707298 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:35:33.707361 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:33.707362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:33.707387 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:33.707479 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:35:33.707752 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.708375 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.709258 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.709522 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.709585 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:33.709723 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:35:33.709853 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:35:33.709921 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:33.709959 (+ 38us) server_negotiation.cc:299] Negotiation successful
0618 23:35:33.709979 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293}
I20260618 23:35:33.712895 17677 raft_consensus.cc:1261] T 6b6f5edfafc842b9be921ece2ffc7573 P 4cf154ba57c64e0c93ec1ff053467f2e [term 1 FOLLOWER]: Refusing update from remote peer d4b02bb2d1b941fea8965627ccd0e587: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:35:33.712877 17543 raft_consensus.cc:1261] T 6b6f5edfafc842b9be921ece2ffc7573 P 277339c7093f4d73a767d2102c851568 [term 1 FOLLOWER]: Refusing update from remote peer d4b02bb2d1b941fea8965627ccd0e587: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:35:33.713440 17884 consensus_queue.cc:1035] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cf154ba57c64e0c93ec1ff053467f2e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:35:33.713636 17880 consensus_queue.cc:1035] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587 [LEADER]: Connected to new peer: Peer: permanent_uuid: "277339c7093f4d73a767d2102c851568" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:35:33.716523 17889 mvcc.cc:205] Tried to move back new op lower bound from 7298358205283602432 to 7298358205231415296. Current Snapshot: MvccSnapshot[applied={T|T < 7298358205283602432}]
I20260618 23:35:33.716961 17888 mvcc.cc:205] Tried to move back new op lower bound from 7298358205283602432 to 7298358205231415296. Current Snapshot: MvccSnapshot[applied={T|T < 7298358205283602432}]
I20260618 23:35:33.719175 17398 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:35:33.719333 17398 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:35:33.720614 17398 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 087b99b40dde4dc8a1abe2f5c027a7e3: Sending DeleteTablet for 3 replicas of tablet 6b6f5edfafc842b9be921ece2ffc7573
I20260618 23:35:33.721073 17657 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b6f5edfafc842b9be921ece2ffc7573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:35:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55970
I20260618 23:35:33.721385 17896 tablet_replica.cc:330] T 6b6f5edfafc842b9be921ece2ffc7573 P 4cf154ba57c64e0c93ec1ff053467f2e: stopping tablet replica
I20260618 23:35:33.721398 17791 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b6f5edfafc842b9be921ece2ffc7573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:35:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50460
I20260618 23:35:33.721561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 17460
I20260618 23:35:33.721669 17523 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b6f5edfafc842b9be921ece2ffc7573 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:35:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40628
I20260618 23:35:33.722357 17897 tablet_replica.cc:330] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587: stopping tablet replica
I20260618 23:35:33.722476 17897 raft_consensus.cc:2229] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:35:33.722617 17897 raft_consensus.cc:2258] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:35:33.723011 17896 raft_consensus.cc:2229] T 6b6f5edfafc842b9be921ece2ffc7573 P 4cf154ba57c64e0c93ec1ff053467f2e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:35:33.723166 17896 raft_consensus.cc:2258] T 6b6f5edfafc842b9be921ece2ffc7573 P 4cf154ba57c64e0c93ec1ff053467f2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:35:33.723742 17896 ts_tablet_manager.cc:1905] T 6b6f5edfafc842b9be921ece2ffc7573 P 4cf154ba57c64e0c93ec1ff053467f2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:35:33.724121 17897 ts_tablet_manager.cc:1905] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:35:33.725710 17896 ts_tablet_manager.cc:1918] T 6b6f5edfafc842b9be921ece2ffc7573 P 4cf154ba57c64e0c93ec1ff053467f2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:35:33.725821 17896 log.cc:1196] T 6b6f5edfafc842b9be921ece2ffc7573 P 4cf154ba57c64e0c93ec1ff053467f2e: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/6b6f5edfafc842b9be921ece2ffc7573
I20260618 23:35:33.726121 17896 ts_tablet_manager.cc:1939] T 6b6f5edfafc842b9be921ece2ffc7573 P 4cf154ba57c64e0c93ec1ff053467f2e: Deleting consensus metadata
I20260618 23:35:33.726363 17897 ts_tablet_manager.cc:1918] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:35:33.726464 17897 log.cc:1196] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/6b6f5edfafc842b9be921ece2ffc7573
I20260618 23:35:33.726694 17382 catalog_manager.cc:4841] TS 4cf154ba57c64e0c93ec1ff053467f2e (127.0.98.2:44653): tablet 6b6f5edfafc842b9be921ece2ffc7573 (table test-table [id=df32460d111549a896063bd216081629]) successfully deleted
I20260618 23:35:33.726768 17897 ts_tablet_manager.cc:1939] T 6b6f5edfafc842b9be921ece2ffc7573 P d4b02bb2d1b941fea8965627ccd0e587: Deleting consensus metadata
I20260618 23:35:33.727281 17385 catalog_manager.cc:4841] TS d4b02bb2d1b941fea8965627ccd0e587 (127.0.98.3:44875): tablet 6b6f5edfafc842b9be921ece2ffc7573 (table test-table [id=df32460d111549a896063bd216081629]) successfully deleted
W20260618 23:35:33.728245 17385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45557 (error 108)
I20260618 23:35:33.728950 17868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:33.728793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45557 (local address 127.0.0.1:40640)
0618 23:35:33.728860 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:33.728903 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260618 23:35:33.728972 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 17594
W20260618 23:35:33.730242 17385 catalog_manager.cc:4584] TS 277339c7093f4d73a767d2102c851568 (127.0.98.1:45557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b6f5edfafc842b9be921ece2ffc7573: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45557: connect: Connection refused (error 111)
I20260618 23:35:33.735018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 17728
I20260618 23:35:33.741191 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 17366
2026-06-18T23:35:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2066 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-18T23:35:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:35:33Z Disabled control of system clock
I20260618 23:35:33.772606 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32863
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 23:35:33.856832 17902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:33.857091 17902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:33.857115 17902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:33.858335 17902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:35:33.858392 17902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:33.858407 17902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:33.858420 17902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:35:33.858433 17902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:35:33.859819 17902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32863
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:33.860062 17902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:33.860328 17902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:33.866202 17910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:33.866178 17908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:33.866230 17907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:33.866421 17902 server_base.cc:1034] running on GCE node
I20260618 23:35:33.866904 17902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:33.867309 17902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:33.868640 17902 hybrid_clock.cc:648] HybridClock initialized: now 1781825733868608 us; error 50 us; skew 500 ppm
I20260618 23:35:33.870366 17902 webserver.cc:460] Webserver started at http://127.0.98.62:36177/ using document root <none> and password file <none>
I20260618 23:35:33.870666 17902 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:33.870745 17902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:33.870909 17902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:33.872259 17902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "41bbbaa08cea4395b0aa3ace842aa705"
format_stamp: "Formatted at 2026-06-18 23:35:33 on dist-test-slave-6224"
server_key: "8be6e91ba5dddf6d68771b242902327a"
server_key_iv: "06637b16eb5e6a9b39eb23129e8e3b53"
server_key_version: "encryptionkey@0"
I20260618 23:35:33.872679 17902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "41bbbaa08cea4395b0aa3ace842aa705"
format_stamp: "Formatted at 2026-06-18 23:35:33 on dist-test-slave-6224"
server_key: "8be6e91ba5dddf6d68771b242902327a"
server_key_iv: "06637b16eb5e6a9b39eb23129e8e3b53"
server_key_version: "encryptionkey@0"
I20260618 23:35:33.874323 17902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:35:33.875545 17916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:33.875921 17902 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:33.876056 17902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "41bbbaa08cea4395b0aa3ace842aa705"
format_stamp: "Formatted at 2026-06-18 23:35:33 on dist-test-slave-6224"
server_key: "8be6e91ba5dddf6d68771b242902327a"
server_key_iv: "06637b16eb5e6a9b39eb23129e8e3b53"
server_key_version: "encryptionkey@0"
I20260618 23:35:33.876194 17902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:33.890020 17902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:33.890483 17902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:33.890668 17902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:33.895879 17902 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32863
I20260618 23:35:33.895948 17968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32863 every 8 connection(s)
I20260618 23:35:33.896416 17902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:35:33.897722 17969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:33.898722 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 17902
I20260618 23:35:33.898881 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:35:33.899122 392 external_mini_cluster.cc:1428] Setting key a1ccc3318ff7f547425d310e03281850
I20260618 23:35:33.900904 17969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705: Bootstrap starting.
I20260618 23:35:33.901615 17969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:33.901954 17969 log.cc:824] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:33.902709 17969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705: No bootstrap required, opened a new log
I20260618 23:35:33.904764 17969 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bbbaa08cea4395b0aa3ace842aa705" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32863 } }
I20260618 23:35:33.904930 17969 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:33.905016 17969 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41bbbaa08cea4395b0aa3ace842aa705, State: Initialized, Role: FOLLOWER
I20260618 23:35:33.905193 17969 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bbbaa08cea4395b0aa3ace842aa705" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32863 } }
I20260618 23:35:33.905268 17969 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:35:33.905290 17969 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:35:33.905349 17969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:33.905999 17969 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bbbaa08cea4395b0aa3ace842aa705" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32863 } }
I20260618 23:35:33.906122 17969 leader_election.cc:302] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41bbbaa08cea4395b0aa3ace842aa705; no voters:
I20260618 23:35:33.906255 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:33.899972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54748 (local address 127.0.98.62:32863)
0618 23:35:33.900320 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:33.900331 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:33.900381 (+ 50us) server_negotiation.cc:407] Connection header received
0618 23:35:33.901054 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:33.901066 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:33.901176 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:33.901374 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:33.901651 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.902768 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.903717 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:33.904211 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:33.904373 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:33.905181 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:33.905191 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:33.905191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:33.905487 (+ 296us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:33.905536 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:33.905550 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:33.905636 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:33.905717 (+ 81us) server_negotiation.cc:299] Negotiation successful
0618 23:35:33.905928 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":114,"threads_started":1}
I20260618 23:35:33.906376 17969 leader_election.cc:288] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:35:33.906502 17974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:33.906641 17974 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705 [term 1 LEADER]: Becoming Leader. State: Replica: 41bbbaa08cea4395b0aa3ace842aa705, State: Running, Role: LEADER
I20260618 23:35:33.906754 17969 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:35:33.906735 17974 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bbbaa08cea4395b0aa3ace842aa705" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32863 } }
I20260618 23:35:33.907116 17974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41bbbaa08cea4395b0aa3ace842aa705. Latest consensus state: current_term: 1 leader_uuid: "41bbbaa08cea4395b0aa3ace842aa705" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bbbaa08cea4395b0aa3ace842aa705" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32863 } } }
I20260618 23:35:33.907195 17974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:33.907490 17979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:35:33.907490 17975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41bbbaa08cea4395b0aa3ace842aa705" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41bbbaa08cea4395b0aa3ace842aa705" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32863 } } }
I20260618 23:35:33.907843 17975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:33.908149 17979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:35:33.909987 17979 catalog_manager.cc:1347] Generated new cluster ID: 6c264ed7fa1941c1a9d01032cbda8c84
I20260618 23:35:33.910065 17979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:35:33.919001 17979 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:35:33.919759 17979 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:35:33.929107 17979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705: Generated new TSK 0
I20260618 23:35:33.929374 17979 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:35:33.935467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32863
--builtin_ntp_servers=127.0.98.20:37695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 23:35:34.031152 17993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:34.031432 17993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:34.031461 17993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:34.032924 17993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:34.032979 17993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:34.033010 17993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:35:34.034580 17993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:34.034917 17993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:34.035312 17993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:34.041577 17999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:34.041560 18001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:34.041718 17993 server_base.cc:1034] running on GCE node
W20260618 23:35:34.041596 17998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:34.042299 17993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:34.042635 17993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:34.043835 17993 hybrid_clock.cc:648] HybridClock initialized: now 1781825734043807 us; error 43 us; skew 500 ppm
I20260618 23:35:34.045934 17993 webserver.cc:460] Webserver started at http://127.0.98.1:43821/ using document root <none> and password file <none>
I20260618 23:35:34.046281 17993 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:34.046343 17993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:34.046509 17993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:34.047977 17993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "031b684d6dfa43b2a08d292edb20aaf7"
format_stamp: "Formatted at 2026-06-18 23:35:34 on dist-test-slave-6224"
server_key: "3c85d520aa724f32124500589ab0a8e9"
server_key_iv: "3ce77e04371dbd77b2e41e59993a22ce"
server_key_version: "encryptionkey@0"
I20260618 23:35:34.048415 17993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "031b684d6dfa43b2a08d292edb20aaf7"
format_stamp: "Formatted at 2026-06-18 23:35:34 on dist-test-slave-6224"
server_key: "3c85d520aa724f32124500589ab0a8e9"
server_key_iv: "3ce77e04371dbd77b2e41e59993a22ce"
server_key_version: "encryptionkey@0"
I20260618 23:35:34.050132 17993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:35:34.051208 18007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:34.051641 17993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:34.051764 17993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "031b684d6dfa43b2a08d292edb20aaf7"
format_stamp: "Formatted at 2026-06-18 23:35:34 on dist-test-slave-6224"
server_key: "3c85d520aa724f32124500589ab0a8e9"
server_key_iv: "3ce77e04371dbd77b2e41e59993a22ce"
server_key_version: "encryptionkey@0"
I20260618 23:35:34.051867 17993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:34.066268 17993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:34.066766 17993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:34.066941 17993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:34.067287 17993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:35:34.067795 17993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:35:34.067854 17993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:34.067906 17993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:35:34.067934 17993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:34.076153 17993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38181
I20260618 23:35:34.076197 18120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38181 every 8 connection(s)
I20260618 23:35:34.076663 17993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:35:34.082886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 17993
I20260618 23:35:34.083012 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:35:34.083376 392 external_mini_cluster.cc:1428] Setting key 16afff0a80586518386f2a72b09a82c3
I20260618 23:35:34.084386 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:34.077595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55761 (local address 127.0.98.62:32863)
0618 23:35:34.077688 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:34.077690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:34.078099 (+ 409us) server_negotiation.cc:407] Connection header received
0618 23:35:34.078785 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:34.078787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:34.078808 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:34.078895 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:34.079962 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.080575 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.081803 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.082019 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.082094 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:34.083797 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:34.083799 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:34.083800 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:34.083807 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:34.083838 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:34.083844 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:34.083908 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:34.084184 (+ 276us) server_negotiation.cc:299] Negotiation successful
0618 23:35:34.084230 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 23:35:34.084810 18123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:34.077597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32863 (local address 127.0.98.1:55761)
0618 23:35:34.077914 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:34.077963 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:34.078635 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:34.078998 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:34.079002 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:34.079013 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:35:34.079827 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:34.079834 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.080722 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.080723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:34.081688 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:34.081691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.082220 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.082221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:34.082372 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:34.083531 (+ 1159us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:35:34.083532 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:35:34.083626 (+ 94us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:35:34.083671 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:35:34.083989 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:35:34.084012 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:35:34.084017 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:35:34.084149 (+ 132us) client_negotiation.cc:240] Negotiation successful
0618 23:35:34.084337 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":127,"threads_started":1}
I20260618 23:35:34.085117 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32863
--builtin_ntp_servers=127.0.98.20:37695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 23:35:34.085511 18121 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32863
I20260618 23:35:34.085667 18121 heartbeater.cc:464] Registering TS with master...
I20260618 23:35:34.085918 18121 heartbeater.cc:511] Master 127.0.98.62:32863 requested a full tablet report, sending...
I20260618 23:35:34.086694 17933 ts_manager.cc:194] Registered new tserver with Master: 031b684d6dfa43b2a08d292edb20aaf7 (127.0.98.1:38181)
I20260618 23:35:34.087451 17933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:55761
W20260618 23:35:34.179064 18124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:34.179328 18124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:34.179354 18124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:34.180707 18124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:34.180768 18124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:34.180802 18124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:35:34.182361 18124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:34.182648 18124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:34.182960 18124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:34.188972 18132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:34.188997 18129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:34.189127 18130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:34.189419 18124 server_base.cc:1034] running on GCE node
I20260618 23:35:34.189675 18124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:34.189950 18124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:34.191114 18124 hybrid_clock.cc:648] HybridClock initialized: now 1781825734191094 us; error 37 us; skew 500 ppm
I20260618 23:35:34.192678 18124 webserver.cc:460] Webserver started at http://127.0.98.2:38781/ using document root <none> and password file <none>
I20260618 23:35:34.193001 18124 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:34.193056 18124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:34.193208 18124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:34.194406 18124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "31b34b6ce1a74163b492a0c6c6aedc01"
format_stamp: "Formatted at 2026-06-18 23:35:34 on dist-test-slave-6224"
server_key: "08d3a01b137f9cae7c043d5bac8c42e5"
server_key_iv: "e45cb8b391eb616c9e8d98ee4f0d0bd2"
server_key_version: "encryptionkey@0"
I20260618 23:35:34.194802 18124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "31b34b6ce1a74163b492a0c6c6aedc01"
format_stamp: "Formatted at 2026-06-18 23:35:34 on dist-test-slave-6224"
server_key: "08d3a01b137f9cae7c043d5bac8c42e5"
server_key_iv: "e45cb8b391eb616c9e8d98ee4f0d0bd2"
server_key_version: "encryptionkey@0"
I20260618 23:35:34.196525 18124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:35:34.197618 18138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:34.198029 18124 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:35:34.198179 18124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "31b34b6ce1a74163b492a0c6c6aedc01"
format_stamp: "Formatted at 2026-06-18 23:35:34 on dist-test-slave-6224"
server_key: "08d3a01b137f9cae7c043d5bac8c42e5"
server_key_iv: "e45cb8b391eb616c9e8d98ee4f0d0bd2"
server_key_version: "encryptionkey@0"
I20260618 23:35:34.198294 18124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:34.221899 18124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:34.222276 18124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:34.222447 18124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:34.222729 18124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:35:34.223094 18124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:35:34.223160 18124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:34.223201 18124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:35:34.223242 18124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:34.230068 18124 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43241
I20260618 23:35:34.230157 18251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43241 every 8 connection(s)
I20260618 23:35:34.230562 18124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:35:34.232226 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 18124
I20260618 23:35:34.232360 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:35:34.232581 392 external_mini_cluster.cc:1428] Setting key 22f98a313955b684562e177186a668cf
I20260618 23:35:34.233841 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32863
--builtin_ntp_servers=127.0.98.20:37695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 23:35:34.237833 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:34.231504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47589 (local address 127.0.98.62:32863)
0618 23:35:34.231646 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:34.231650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:34.231891 (+ 241us) server_negotiation.cc:407] Connection header received
0618 23:35:34.232747 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:34.232748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:34.232782 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:34.232875 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:34.233984 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.234537 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.236081 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.236308 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.236399 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:34.237471 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:34.237473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:34.237473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:34.237480 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:34.237515 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:34.237526 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:34.237586 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:34.237689 (+ 103us) server_negotiation.cc:299] Negotiation successful
0618 23:35:34.237728 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260618 23:35:34.238226 18254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:34.231445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32863 (local address 127.0.98.2:47589)
0618 23:35:34.231722 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:34.231766 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:34.232604 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:34.232984 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:34.232989 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:34.233001 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:35:34.233855 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:34.233863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.234652 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.234654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:34.235974 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:34.235976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.236404 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.236405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:34.236563 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:34.237251 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:35:34.237252 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:35:34.237295 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:35:34.237329 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:35:34.237599 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:35:34.237606 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:35:34.237609 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:35:34.237688 (+ 79us) client_negotiation.cc:240] Negotiation successful
0618 23:35:34.237821 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":140,"threads_started":1}
I20260618 23:35:34.238843 18252 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32863
I20260618 23:35:34.238953 18252 heartbeater.cc:464] Registering TS with master...
I20260618 23:35:34.239197 18252 heartbeater.cc:511] Master 127.0.98.62:32863 requested a full tablet report, sending...
I20260618 23:35:34.239707 17933 ts_manager.cc:194] Registered new tserver with Master: 31b34b6ce1a74163b492a0c6c6aedc01 (127.0.98.2:43241)
I20260618 23:35:34.240140 17933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:47589
W20260618 23:35:34.327745 18255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:34.328058 18255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:34.328101 18255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:34.329809 18255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:34.329934 18255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:34.329986 18255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:35:34.331763 18255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:34.332120 18255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:34.332486 18255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:34.338393 18261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:34.338383 18263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:34.338383 18260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:34.338980 18255 server_base.cc:1034] running on GCE node
I20260618 23:35:34.339231 18255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:34.339485 18255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:34.340639 18255 hybrid_clock.cc:648] HybridClock initialized: now 1781825734340625 us; error 43 us; skew 500 ppm
I20260618 23:35:34.342084 18255 webserver.cc:460] Webserver started at http://127.0.98.3:46651/ using document root <none> and password file <none>
I20260618 23:35:34.342355 18255 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:34.342425 18255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:34.342604 18255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:34.343812 18255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "66146d5bcf484822841a5ba963085236"
format_stamp: "Formatted at 2026-06-18 23:35:34 on dist-test-slave-6224"
server_key: "4bb50521f61757b8945aefc3ab16e050"
server_key_iv: "3fb58a1f15b2be3a3f2afcddce0748b6"
server_key_version: "encryptionkey@0"
I20260618 23:35:34.344180 18255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "66146d5bcf484822841a5ba963085236"
format_stamp: "Formatted at 2026-06-18 23:35:34 on dist-test-slave-6224"
server_key: "4bb50521f61757b8945aefc3ab16e050"
server_key_iv: "3fb58a1f15b2be3a3f2afcddce0748b6"
server_key_version: "encryptionkey@0"
I20260618 23:35:34.345611 18255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:35:34.346498 18269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:34.346771 18255 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:34.346881 18255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "66146d5bcf484822841a5ba963085236"
format_stamp: "Formatted at 2026-06-18 23:35:34 on dist-test-slave-6224"
server_key: "4bb50521f61757b8945aefc3ab16e050"
server_key_iv: "3fb58a1f15b2be3a3f2afcddce0748b6"
server_key_version: "encryptionkey@0"
I20260618 23:35:34.346966 18255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:34.364277 18255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:34.364629 18255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:34.364790 18255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:34.365058 18255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:35:34.365438 18255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:35:34.365506 18255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:34.365552 18255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:35:34.365581 18255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:34.372509 18255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39659
I20260618 23:35:34.372699 18382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39659 every 8 connection(s)
I20260618 23:35:34.372999 18255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:35:34.379624 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:34.373990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51981 (local address 127.0.98.62:32863)
0618 23:35:34.374111 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:34.374115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:34.374371 (+ 256us) server_negotiation.cc:407] Connection header received
0618 23:35:34.375070 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:34.375072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:34.375086 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:34.375156 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:34.375994 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.376556 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.377672 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.377904 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.378004 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:34.379142 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:34.379144 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:34.379144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:34.379152 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:34.379187 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:34.379193 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:34.379265 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:34.379443 (+ 178us) server_negotiation.cc:299] Negotiation successful
0618 23:35:34.379494 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260618 23:35:34.380003 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:34.373869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32863 (local address 127.0.98.3:51981)
0618 23:35:34.374181 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:34.374236 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:34.374940 (+ 704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:34.375247 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:34.375252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:34.375262 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:35:34.375888 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:34.375893 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.376666 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.376667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:34.377566 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:34.377568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.378015 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.378017 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:34.378185 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:34.378926 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:35:34.378927 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:35:34.378962 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:35:34.378995 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:35:34.379323 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:35:34.379330 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:35:34.379334 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:35:34.379433 (+ 99us) client_negotiation.cc:240] Negotiation successful
0618 23:35:34.379643 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":122,"threads_started":1}
I20260618 23:35:34.380677 18383 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32863
I20260618 23:35:34.380820 18383 heartbeater.cc:464] Registering TS with master...
I20260618 23:35:34.381109 18383 heartbeater.cc:511] Master 127.0.98.62:32863 requested a full tablet report, sending...
I20260618 23:35:34.381321 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 18255
I20260618 23:35:34.381474 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:35:34.381704 392 external_mini_cluster.cc:1428] Setting key 619f2f0bdc3d7d92be70c5e9813cca7a
I20260618 23:35:34.381721 17933 ts_manager.cc:194] Registered new tserver with Master: 66146d5bcf484822841a5ba963085236 (127.0.98.3:39659)
I20260618 23:35:34.382359 17933 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:51981
I20260618 23:35:34.382820 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:35:34.387640 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:34.384519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54752 (local address 127.0.98.62:32863)
0618 23:35:34.384584 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:34.384586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:34.384598 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:35:34.384663 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:34.384664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:34.384673 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:34.384740 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:34.385006 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.385525 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.386320 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.386524 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.386596 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:34.387301 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:34.387303 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:34.387303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:34.387309 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:34.387327 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:34.387331 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:34.387386 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:34.387459 (+ 73us) server_negotiation.cc:299] Negotiation successful
0618 23:35:34.387487 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:35:34.389694 17932 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:35:34.391677 17932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:35:34.399420 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:34.394872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59704 (local address 127.0.98.3:39659)
0618 23:35:34.395305 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:34.395309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:34.395320 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:35:34.395359 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:34.395360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:34.395374 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:34.395450 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:34.395763 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.397008 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.398298 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.398913 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.399022 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:34.399139 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:34.399233 (+ 94us) server_negotiation.cc:299] Negotiation successful
0618 23:35:34.399287 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":115,"threads_started":1}
I20260618 23:35:34.399421 18392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:34.394660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39659 (local address 127.0.0.1:59704)
0618 23:35:34.395079 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:34.395126 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:34.395212 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:34.395527 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:34.395528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:34.395530 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:34.395672 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:34.395676 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.397116 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.397117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:34.398180 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:34.398182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.399021 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.399023 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:34.399105 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:34.399130 (+ 25us) client_negotiation.cc:769] Sending connection context
0618 23:35:34.399212 (+ 82us) client_negotiation.cc:240] Negotiation successful
0618 23:35:34.399288 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":22,"thread_start_us":112,"threads_started":1}
I20260618 23:35:34.399721 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:34.394664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43241 (local address 127.0.0.1:44068)
0618 23:35:34.395104 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:34.395142 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:34.395212 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:34.395514 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:34.395518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:34.395521 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:34.395672 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:34.395675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.397120 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.397121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:34.398300 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:34.398302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.399318 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.399318 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:34.399434 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:34.399438 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:35:34.399495 (+ 57us) client_negotiation.cc:240] Negotiation successful
0618 23:35:34.399590 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":51,"thread_start_us":122,"threads_started":1}
I20260618 23:35:34.400043 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:34.395002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44068 (local address 127.0.98.2:43241)
0618 23:35:34.395208 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:34.395214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:34.395227 (+ 13us) server_negotiation.cc:407] Connection header received
0618 23:35:34.395337 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:34.395339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:34.395357 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:34.395431 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:34.395763 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.397010 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.398402 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.399088 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.399160 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:34.399267 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:34.399907 (+ 640us) server_negotiation.cc:299] Negotiation successful
0618 23:35:34.399959 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":91,"threads_started":1}
I20260618 23:35:34.400152 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:34.394863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47054 (local address 127.0.98.1:38181)
0618 23:35:34.395113 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:34.395120 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:34.395458 (+ 338us) server_negotiation.cc:407] Connection header received
0618 23:35:34.395509 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:34.395512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:34.395528 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:34.395650 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:34.395826 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.396587 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.398175 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.399076 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.399160 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:34.399267 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:34.400005 (+ 738us) server_negotiation.cc:299] Negotiation successful
0618 23:35:34.400060 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":103,"threads_started":1}
I20260618 23:35:34.400368 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:34.394664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38181 (local address 127.0.0.1:47054)
0618 23:35:34.395337 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:34.395346 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:34.395400 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:34.395660 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:34.395661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:34.395663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:34.395735 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:34.395738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.396689 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.396690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:34.398039 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:34.398042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.399316 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.399317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:34.399411 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:34.399429 (+ 18us) client_negotiation.cc:769] Sending connection context
0618 23:35:34.400241 (+ 812us) client_negotiation.cc:240] Negotiation successful
0618 23:35:34.400298 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":5,"thread_start_us":75,"threads_started":1}
I20260618 23:35:34.400739 18317 tablet_service.cc:1467] Processing CreateTablet for tablet 71ca632b981a4f0aa56c1da18279996c (DEFAULT_TABLE table=test-table [id=1c58f6fe9a234fa2b8ae50ebef5e0be6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:35:34.400846 18186 tablet_service.cc:1467] Processing CreateTablet for tablet 71ca632b981a4f0aa56c1da18279996c (DEFAULT_TABLE table=test-table [id=1c58f6fe9a234fa2b8ae50ebef5e0be6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:35:34.401243 18317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71ca632b981a4f0aa56c1da18279996c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:34.401274 18186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71ca632b981a4f0aa56c1da18279996c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:34.404170 18055 tablet_service.cc:1467] Processing CreateTablet for tablet 71ca632b981a4f0aa56c1da18279996c (DEFAULT_TABLE table=test-table [id=1c58f6fe9a234fa2b8ae50ebef5e0be6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:35:34.404439 18397 tablet_bootstrap.cc:492] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01: Bootstrap starting.
I20260618 23:35:34.404611 18055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71ca632b981a4f0aa56c1da18279996c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:34.404930 18398 tablet_bootstrap.cc:492] T 71ca632b981a4f0aa56c1da18279996c P 66146d5bcf484822841a5ba963085236: Bootstrap starting.
I20260618 23:35:34.405151 18397 tablet_bootstrap.cc:654] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:34.405479 18397 log.cc:824] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:34.405629 18398 tablet_bootstrap.cc:654] T 71ca632b981a4f0aa56c1da18279996c P 66146d5bcf484822841a5ba963085236: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:34.405931 18398 log.cc:824] T 71ca632b981a4f0aa56c1da18279996c P 66146d5bcf484822841a5ba963085236: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:34.406479 18397 tablet_bootstrap.cc:492] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01: No bootstrap required, opened a new log
I20260618 23:35:34.406545 18397 ts_tablet_manager.cc:1397] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:35:34.406747 18398 tablet_bootstrap.cc:492] T 71ca632b981a4f0aa56c1da18279996c P 66146d5bcf484822841a5ba963085236: No bootstrap required, opened a new log
I20260618 23:35:34.406798 18398 ts_tablet_manager.cc:1397] T 71ca632b981a4f0aa56c1da18279996c P 66146d5bcf484822841a5ba963085236: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:35:34.408913 18401 tablet_bootstrap.cc:492] T 71ca632b981a4f0aa56c1da18279996c P 031b684d6dfa43b2a08d292edb20aaf7: Bootstrap starting.
I20260618 23:35:34.409614 18401 tablet_bootstrap.cc:654] T 71ca632b981a4f0aa56c1da18279996c P 031b684d6dfa43b2a08d292edb20aaf7: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:34.409720 18398 raft_consensus.cc:348] T 71ca632b981a4f0aa56c1da18279996c P 66146d5bcf484822841a5ba963085236 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031b684d6dfa43b2a08d292edb20aaf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38181 } } peers { permanent_uuid: "66146d5bcf484822841a5ba963085236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39659 } } peers { permanent_uuid: "31b34b6ce1a74163b492a0c6c6aedc01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43241 } }
I20260618 23:35:34.409931 18398 raft_consensus.cc:374] T 71ca632b981a4f0aa56c1da18279996c P 66146d5bcf484822841a5ba963085236 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:34.409968 18398 raft_consensus.cc:729] T 71ca632b981a4f0aa56c1da18279996c P 66146d5bcf484822841a5ba963085236 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66146d5bcf484822841a5ba963085236, State: Initialized, Role: FOLLOWER
I20260618 23:35:34.409999 18401 log.cc:824] T 71ca632b981a4f0aa56c1da18279996c P 031b684d6dfa43b2a08d292edb20aaf7: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:34.410140 18398 consensus_queue.cc:260] T 71ca632b981a4f0aa56c1da18279996c P 66146d5bcf484822841a5ba963085236 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031b684d6dfa43b2a08d292edb20aaf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38181 } } peers { permanent_uuid: "66146d5bcf484822841a5ba963085236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39659 } } peers { permanent_uuid: "31b34b6ce1a74163b492a0c6c6aedc01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43241 } }
I20260618 23:35:34.410104 18397 raft_consensus.cc:348] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031b684d6dfa43b2a08d292edb20aaf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38181 } } peers { permanent_uuid: "66146d5bcf484822841a5ba963085236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39659 } } peers { permanent_uuid: "31b34b6ce1a74163b492a0c6c6aedc01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43241 } }
I20260618 23:35:34.410241 18397 raft_consensus.cc:374] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:34.410303 18397 raft_consensus.cc:729] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b34b6ce1a74163b492a0c6c6aedc01, State: Initialized, Role: FOLLOWER
I20260618 23:35:34.410449 18398 ts_tablet_manager.cc:1428] T 71ca632b981a4f0aa56c1da18279996c P 66146d5bcf484822841a5ba963085236: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260618 23:35:34.410426 18397 consensus_queue.cc:260] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031b684d6dfa43b2a08d292edb20aaf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38181 } } peers { permanent_uuid: "66146d5bcf484822841a5ba963085236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39659 } } peers { permanent_uuid: "31b34b6ce1a74163b492a0c6c6aedc01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43241 } }
I20260618 23:35:34.410600 18383 heartbeater.cc:503] Master 127.0.98.62:32863 was elected leader, sending a full tablet report...
I20260618 23:35:34.410668 18397 ts_tablet_manager.cc:1428] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260618 23:35:34.410784 18252 heartbeater.cc:503] Master 127.0.98.62:32863 was elected leader, sending a full tablet report...
I20260618 23:35:34.410966 18401 tablet_bootstrap.cc:492] T 71ca632b981a4f0aa56c1da18279996c P 031b684d6dfa43b2a08d292edb20aaf7: No bootstrap required, opened a new log
I20260618 23:35:34.411041 18401 ts_tablet_manager.cc:1397] T 71ca632b981a4f0aa56c1da18279996c P 031b684d6dfa43b2a08d292edb20aaf7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:35:34.413254 18401 raft_consensus.cc:348] T 71ca632b981a4f0aa56c1da18279996c P 031b684d6dfa43b2a08d292edb20aaf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031b684d6dfa43b2a08d292edb20aaf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38181 } } peers { permanent_uuid: "66146d5bcf484822841a5ba963085236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39659 } } peers { permanent_uuid: "31b34b6ce1a74163b492a0c6c6aedc01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43241 } }
I20260618 23:35:34.413443 18401 raft_consensus.cc:374] T 71ca632b981a4f0aa56c1da18279996c P 031b684d6dfa43b2a08d292edb20aaf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:34.413496 18401 raft_consensus.cc:729] T 71ca632b981a4f0aa56c1da18279996c P 031b684d6dfa43b2a08d292edb20aaf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031b684d6dfa43b2a08d292edb20aaf7, State: Initialized, Role: FOLLOWER
I20260618 23:35:34.413636 18401 consensus_queue.cc:260] T 71ca632b981a4f0aa56c1da18279996c P 031b684d6dfa43b2a08d292edb20aaf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031b684d6dfa43b2a08d292edb20aaf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38181 } } peers { permanent_uuid: "66146d5bcf484822841a5ba963085236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39659 } } peers { permanent_uuid: "31b34b6ce1a74163b492a0c6c6aedc01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43241 } }
I20260618 23:35:34.413945 18401 ts_tablet_manager.cc:1428] T 71ca632b981a4f0aa56c1da18279996c P 031b684d6dfa43b2a08d292edb20aaf7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:35:34.414068 18121 heartbeater.cc:503] Master 127.0.98.62:32863 was elected leader, sending a full tablet report...
W20260618 23:35:34.481294 18253 tablet.cc:2369] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:35:34.549175 18404 raft_consensus.cc:482] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:35:34.549376 18404 raft_consensus.cc:504] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031b684d6dfa43b2a08d292edb20aaf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38181 } } peers { permanent_uuid: "66146d5bcf484822841a5ba963085236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39659 } } peers { permanent_uuid: "31b34b6ce1a74163b492a0c6c6aedc01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43241 } }
I20260618 23:35:34.549768 18404 leader_election.cc:288] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031b684d6dfa43b2a08d292edb20aaf7 (127.0.98.1:38181), 66146d5bcf484822841a5ba963085236 (127.0.98.3:39659)
I20260618 23:35:34.553187 18406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:34.550064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38181 (local address 127.0.98.2:49271)
0618 23:35:34.550313 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:34.550324 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:34.550380 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:34.550513 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:34.550514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:34.550515 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:34.550594 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:34.550597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.551383 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.551385 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:34.552326 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:34.552329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.552888 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.552890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:34.552969 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:34.552973 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:35:34.553049 (+ 76us) client_negotiation.cc:240] Negotiation successful
0618 23:35:34.553095 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":169,"threads_started":1}
I20260618 23:35:34.553172 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:34.550142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49271 (local address 127.0.98.1:38181)
0618 23:35:34.550190 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:34.550192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:34.550375 (+ 183us) server_negotiation.cc:407] Connection header received
0618 23:35:34.550438 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:34.550439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:34.550451 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:34.550495 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:34.550672 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.551283 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.552432 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.552790 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.552887 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:34.552898 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:34.553054 (+ 156us) server_negotiation.cc:299] Negotiation successful
0618 23:35:34.553092 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:35:34.553407 18254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:34.550054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39659 (local address 127.0.98.2:50005)
0618 23:35:34.550172 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:34.550190 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:34.550263 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:34.550583 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:34.550585 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:34.550587 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:34.550700 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:34.550702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.551747 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.551748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:34.552519 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:34.552521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.553101 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.553102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:34.553181 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:34.553189 (+ 8us) client_negotiation.cc:769] Sending connection context
0618 23:35:34.553263 (+ 74us) client_negotiation.cc:240] Negotiation successful
0618 23:35:34.553314 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260618 23:35:34.553407 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:34.550310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50005 (local address 127.0.98.3:39659)
0618 23:35:34.550405 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:34.550408 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:34.550424 (+ 16us) server_negotiation.cc:407] Connection header received
0618 23:35:34.550470 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:34.550471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:34.550480 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:34.550571 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:34.550792 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.551628 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.552623 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.553009 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.553102 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:34.553124 (+ 22us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:34.553269 (+ 145us) server_negotiation.cc:299] Negotiation successful
0618 23:35:34.553315 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 23:35:34.553912 18337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ca632b981a4f0aa56c1da18279996c" candidate_uuid: "31b34b6ce1a74163b492a0c6c6aedc01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66146d5bcf484822841a5ba963085236" is_pre_election: true
I20260618 23:35:34.553967 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ca632b981a4f0aa56c1da18279996c" candidate_uuid: "31b34b6ce1a74163b492a0c6c6aedc01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031b684d6dfa43b2a08d292edb20aaf7" is_pre_election: true
I20260618 23:35:34.554142 18075 raft_consensus.cc:2454] T 71ca632b981a4f0aa56c1da18279996c P 031b684d6dfa43b2a08d292edb20aaf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b34b6ce1a74163b492a0c6c6aedc01 in term 0.
I20260618 23:35:34.554142 18337 raft_consensus.cc:2454] T 71ca632b981a4f0aa56c1da18279996c P 66146d5bcf484822841a5ba963085236 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31b34b6ce1a74163b492a0c6c6aedc01 in term 0.
I20260618 23:35:34.554469 18142 leader_election.cc:302] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031b684d6dfa43b2a08d292edb20aaf7, 31b34b6ce1a74163b492a0c6c6aedc01; no voters:
I20260618 23:35:34.554627 18404 raft_consensus.cc:2781] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:35:34.554692 18404 raft_consensus.cc:482] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:35:34.554718 18404 raft_consensus.cc:3037] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:34.555786 18404 raft_consensus.cc:504] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031b684d6dfa43b2a08d292edb20aaf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38181 } } peers { permanent_uuid: "66146d5bcf484822841a5ba963085236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39659 } } peers { permanent_uuid: "31b34b6ce1a74163b492a0c6c6aedc01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43241 } }
I20260618 23:35:34.555980 18404 leader_election.cc:288] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01 [CANDIDATE]: Term 1 election: Requested vote from peers 031b684d6dfa43b2a08d292edb20aaf7 (127.0.98.1:38181), 66146d5bcf484822841a5ba963085236 (127.0.98.3:39659)
I20260618 23:35:34.556236 18075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ca632b981a4f0aa56c1da18279996c" candidate_uuid: "31b34b6ce1a74163b492a0c6c6aedc01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031b684d6dfa43b2a08d292edb20aaf7"
I20260618 23:35:34.556375 18075 raft_consensus.cc:3037] T 71ca632b981a4f0aa56c1da18279996c P 031b684d6dfa43b2a08d292edb20aaf7 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:34.557337 18075 raft_consensus.cc:2454] T 71ca632b981a4f0aa56c1da18279996c P 031b684d6dfa43b2a08d292edb20aaf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b34b6ce1a74163b492a0c6c6aedc01 in term 1.
I20260618 23:35:34.557482 18337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71ca632b981a4f0aa56c1da18279996c" candidate_uuid: "31b34b6ce1a74163b492a0c6c6aedc01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66146d5bcf484822841a5ba963085236"
I20260618 23:35:34.557572 18142 leader_election.cc:302] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031b684d6dfa43b2a08d292edb20aaf7, 31b34b6ce1a74163b492a0c6c6aedc01; no voters:
I20260618 23:35:34.557650 18337 raft_consensus.cc:3037] T 71ca632b981a4f0aa56c1da18279996c P 66146d5bcf484822841a5ba963085236 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:34.557715 18404 raft_consensus.cc:2781] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:34.557983 18404 raft_consensus.cc:686] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01 [term 1 LEADER]: Becoming Leader. State: Replica: 31b34b6ce1a74163b492a0c6c6aedc01, State: Running, Role: LEADER
I20260618 23:35:34.558105 18404 consensus_queue.cc:237] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031b684d6dfa43b2a08d292edb20aaf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38181 } } peers { permanent_uuid: "66146d5bcf484822841a5ba963085236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39659 } } peers { permanent_uuid: "31b34b6ce1a74163b492a0c6c6aedc01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43241 } }
I20260618 23:35:34.558691 18337 raft_consensus.cc:2454] T 71ca632b981a4f0aa56c1da18279996c P 66146d5bcf484822841a5ba963085236 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31b34b6ce1a74163b492a0c6c6aedc01 in term 1.
I20260618 23:35:34.559177 17933 catalog_manager.cc:5495] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31b34b6ce1a74163b492a0c6c6aedc01 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "31b34b6ce1a74163b492a0c6c6aedc01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031b684d6dfa43b2a08d292edb20aaf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66146d5bcf484822841a5ba963085236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31b34b6ce1a74163b492a0c6c6aedc01" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43241 } health_report { overall_health: HEALTHY } } }
W20260618 23:35:34.577724 18122 tablet.cc:2369] T 71ca632b981a4f0aa56c1da18279996c P 031b684d6dfa43b2a08d292edb20aaf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:35:34.588451 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:34.585564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.0.98.2:43241)
0618 23:35:34.585635 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:34.585638 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:34.585649 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:35:34.585698 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:34.585699 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:34.585730 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:34.585834 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:35:34.586011 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.586560 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.587413 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:34.587721 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:34.587788 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:34.587971 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:35:34.588128 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:35:34.588213 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:34.588316 (+ 103us) server_negotiation.cc:299] Negotiation successful
0618 23:35:34.588339 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 23:35:34.591018 18337 raft_consensus.cc:1261] T 71ca632b981a4f0aa56c1da18279996c P 66146d5bcf484822841a5ba963085236 [term 1 FOLLOWER]: Refusing update from remote peer 31b34b6ce1a74163b492a0c6c6aedc01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:35:34.591018 18075 raft_consensus.cc:1261] T 71ca632b981a4f0aa56c1da18279996c P 031b684d6dfa43b2a08d292edb20aaf7 [term 1 FOLLOWER]: Refusing update from remote peer 31b34b6ce1a74163b492a0c6c6aedc01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:35:34.591537 18407 consensus_queue.cc:1035] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031b684d6dfa43b2a08d292edb20aaf7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:35:34.591818 18404 consensus_queue.cc:1035] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66146d5bcf484822841a5ba963085236" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:35:34.596530 18412 mvcc.cc:205] Tried to move back new op lower bound from 7298358208880447488 to 7298358208750706688. Current Snapshot: MvccSnapshot[applied={T|T < 7298358208880447488}]
I20260618 23:35:34.597893 17932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:35:34.598034 17932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:35:34.599133 17932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41bbbaa08cea4395b0aa3ace842aa705: Sending DeleteTablet for 3 replicas of tablet 71ca632b981a4f0aa56c1da18279996c
I20260618 23:35:34.599644 18317 tablet_service.cc:1514] Processing DeleteTablet for tablet 71ca632b981a4f0aa56c1da18279996c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:35:34 UTC) from {username='slave'} at 127.0.0.1:59704
I20260618 23:35:34.599702 18186 tablet_service.cc:1514] Processing DeleteTablet for tablet 71ca632b981a4f0aa56c1da18279996c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:35:34 UTC) from {username='slave'} at 127.0.0.1:44068
I20260618 23:35:34.599673 18055 tablet_service.cc:1514] Processing DeleteTablet for tablet 71ca632b981a4f0aa56c1da18279996c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:35:34 UTC) from {username='slave'} at 127.0.0.1:47054
I20260618 23:35:34.599992 18422 tablet_replica.cc:330] T 71ca632b981a4f0aa56c1da18279996c P 031b684d6dfa43b2a08d292edb20aaf7: stopping tablet replica
I20260618 23:35:34.600075 18420 tablet_replica.cc:330] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01: stopping tablet replica
I20260618 23:35:34.600212 18420 raft_consensus.cc:2229] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:35:34.600320 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 17993
I20260618 23:35:34.601409 18421 tablet_replica.cc:330] T 71ca632b981a4f0aa56c1da18279996c P 66146d5bcf484822841a5ba963085236: stopping tablet replica
I20260618 23:35:34.601565 18421 raft_consensus.cc:2229] T 71ca632b981a4f0aa56c1da18279996c P 66146d5bcf484822841a5ba963085236 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:35:34.601678 18421 raft_consensus.cc:2258] T 71ca632b981a4f0aa56c1da18279996c P 66146d5bcf484822841a5ba963085236 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:35:34.601905 18420 raft_consensus.cc:2258] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:35:34.602746 18420 ts_tablet_manager.cc:1905] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:35:34.605046 18420 ts_tablet_manager.cc:1918] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:35:34.605154 18420 log.cc:1196] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/71ca632b981a4f0aa56c1da18279996c
I20260618 23:35:34.605490 18420 ts_tablet_manager.cc:1939] T 71ca632b981a4f0aa56c1da18279996c P 31b34b6ce1a74163b492a0c6c6aedc01: Deleting consensus metadata
I20260618 23:35:34.606220 17919 catalog_manager.cc:4841] TS 31b34b6ce1a74163b492a0c6c6aedc01 (127.0.98.2:43241): tablet 71ca632b981a4f0aa56c1da18279996c (table test-table [id=1c58f6fe9a234fa2b8ae50ebef5e0be6]) successfully deleted
W20260618 23:35:34.606380 17920 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38181 (error 108)
I20260618 23:35:34.606698 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 18124
I20260618 23:35:34.607014 18421 ts_tablet_manager.cc:1905] T 71ca632b981a4f0aa56c1da18279996c P 66146d5bcf484822841a5ba963085236: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:35:34.607893 18391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:34.607718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38181 (local address 127.0.0.1:47070)
0618 23:35:34.607801 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:34.607851 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260618 23:35:34.608085 17920 catalog_manager.cc:4584] TS 031b684d6dfa43b2a08d292edb20aaf7 (127.0.98.1:38181): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 71ca632b981a4f0aa56c1da18279996c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38181: connect: Connection refused (error 111)
I20260618 23:35:34.609562 18421 ts_tablet_manager.cc:1918] T 71ca632b981a4f0aa56c1da18279996c P 66146d5bcf484822841a5ba963085236: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:35:34.609668 18421 log.cc:1196] T 71ca632b981a4f0aa56c1da18279996c P 66146d5bcf484822841a5ba963085236: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/71ca632b981a4f0aa56c1da18279996c
I20260618 23:35:34.609977 18421 ts_tablet_manager.cc:1939] T 71ca632b981a4f0aa56c1da18279996c P 66146d5bcf484822841a5ba963085236: Deleting consensus metadata
I20260618 23:35:34.610543 17917 catalog_manager.cc:4841] TS 66146d5bcf484822841a5ba963085236 (127.0.98.3:39659): tablet 71ca632b981a4f0aa56c1da18279996c (table test-table [id=1c58f6fe9a234fa2b8ae50ebef5e0be6]) successfully deleted
I20260618 23:35:34.615717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 18255
I20260618 23:35:34.621429 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 17902
2026-06-18T23:35:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (879 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:35:34 dist-test-slave-6224 krb5kdc[18426](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:35:34 dist-test-slave-6224 krb5kdc[18426](info): set up 2 sockets
Jun 18 23:35:34 dist-test-slave-6224 krb5kdc[18426](info): commencing operation
krb5kdc: starting...
W20260618 23:35:35.683923 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:35:35 dist-test-slave-6224 krb5kdc[18426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:35:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:35:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:35:35.823048 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36837
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:35.909866 18439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:35.910116 18439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:35.910140 18439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:35.911381 18439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:35:35.911437 18439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:35.911451 18439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:35.911464 18439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:35:35.911476 18439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:35:35.913094 18439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:35.913367 18439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:35.913659 18439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:35.919281 18445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:35.919284 18444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:35.919616 18439 server_base.cc:1034] running on GCE node
W20260618 23:35:35.919284 18447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:35.920032 18439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:35.920418 18439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:35.921591 18439 hybrid_clock.cc:648] HybridClock initialized: now 1781825735921579 us; error 37 us; skew 500 ppm
Jun 18 23:35:35 dist-test-slave-6224 krb5kdc[18426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825735, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:35.924731 18439 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:35:35.925267 18439 webserver.cc:460] Webserver started at http://127.0.98.62:36275/ using document root <none> and password file <none>
I20260618 23:35:35.925544 18439 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:35.925625 18439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:35.925784 18439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:35.927001 18439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "57a980ed3df64239bb39ba52335fff30"
format_stamp: "Formatted at 2026-06-18 23:35:35 on dist-test-slave-6224"
server_key: "5a27f8dede53c7d823f0273d641cc075"
server_key_iv: "db3356349ecff58caaa5b6e58fc7ee56"
server_key_version: "encryptionkey@0"
I20260618 23:35:35.927397 18439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "57a980ed3df64239bb39ba52335fff30"
format_stamp: "Formatted at 2026-06-18 23:35:35 on dist-test-slave-6224"
server_key: "5a27f8dede53c7d823f0273d641cc075"
server_key_iv: "db3356349ecff58caaa5b6e58fc7ee56"
server_key_version: "encryptionkey@0"
I20260618 23:35:35.929378 18439 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260618 23:35:35.930300 18454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:35.930586 18439 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:35.930737 18439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "57a980ed3df64239bb39ba52335fff30"
format_stamp: "Formatted at 2026-06-18 23:35:35 on dist-test-slave-6224"
server_key: "5a27f8dede53c7d823f0273d641cc075"
server_key_iv: "db3356349ecff58caaa5b6e58fc7ee56"
server_key_version: "encryptionkey@0"
I20260618 23:35:35.930841 18439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:35.957619 18439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:35.960433 18439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:35.960659 18439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:35.966203 18439 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36837
I20260618 23:35:35.966321 18506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36837 every 8 connection(s)
I20260618 23:35:35.966821 18439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:35:35.967936 18507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:35.969514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 18439
I20260618 23:35:35.969700 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:35:35.969947 392 external_mini_cluster.cc:1428] Setting key 700dd2f4f479edf209da0d174e36ea5f
I20260618 23:35:35.970994 18507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30: Bootstrap starting.
I20260618 23:35:35.971792 18507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:35.972124 18507 log.cc:824] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:35.973273 18507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30: No bootstrap required, opened a new log
I20260618 23:35:35.975646 18507 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a980ed3df64239bb39ba52335fff30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } }
I20260618 23:35:35.975827 18507 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:35.975863 18507 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57a980ed3df64239bb39ba52335fff30, State: Initialized, Role: FOLLOWER
I20260618 23:35:35.975993 18507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a980ed3df64239bb39ba52335fff30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } }
I20260618 23:35:35.976078 18507 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:35:35.976130 18507 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:35:35.976173 18507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:35.976958 18507 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a980ed3df64239bb39ba52335fff30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } }
I20260618 23:35:35.977083 18507 leader_election.cc:302] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57a980ed3df64239bb39ba52335fff30; no voters:
I20260618 23:35:35.977263 18507 leader_election.cc:288] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:35:35.977375 18512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:35.977588 18507 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:35:35.977687 18512 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30 [term 1 LEADER]: Becoming Leader. State: Replica: 57a980ed3df64239bb39ba52335fff30, State: Running, Role: LEADER
I20260618 23:35:35.977861 18512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a980ed3df64239bb39ba52335fff30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } }
Jun 18 23:35:35 dist-test-slave-6224 krb5kdc[18426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:35:35.979419 18513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57a980ed3df64239bb39ba52335fff30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a980ed3df64239bb39ba52335fff30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } } }
I20260618 23:35:35.979678 18513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:35.979440 18514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57a980ed3df64239bb39ba52335fff30. Latest consensus state: current_term: 1 leader_uuid: "57a980ed3df64239bb39ba52335fff30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57a980ed3df64239bb39ba52335fff30" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36837 } } }
I20260618 23:35:35.979950 18514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:35.979981 18527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:35:35.980566 18527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:35:35.982813 18527 catalog_manager.cc:1347] Generated new cluster ID: d36546d9a3b64637b6c4e405188d99a7
I20260618 23:35:35.982874 18527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:35:35.984158 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:35.970778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47098 (local address 127.0.98.62:36837)
0618 23:35:35.971071 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:35.971083 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:35.971122 (+ 39us) server_negotiation.cc:407] Connection header received
0618 23:35:35.971809 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:35.971818 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:35.971886 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:35.972078 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:35.973109 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:35.974499 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:35.975422 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:35.976064 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:35.976188 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:35.979753 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:35.979758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:35.979759 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:35.979776 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:35.981989 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:35.982468 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:35.982471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:35.982471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:35.982585 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:35.982792 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:35.982793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:35.982793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:35.983141 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:35.983272 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:35.983763 (+ 491us) server_negotiation.cc:299] Negotiation successful
0618 23:35:35.983911 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":154,"threads_started":1}
I20260618 23:35:35.995656 18527 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:35:35.996280 18527 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:35:36.006029 18527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30: Generated new TSK 0
I20260618 23:35:36.006254 18527 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260618 23:35:36.039062 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36837
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:36.124923 18533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:36.125133 18533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:36.125159 18533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:36.126298 18533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:36.126345 18533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:36.126372 18533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:35:36.127867 18533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:36.128113 18533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:36.128365 18533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:36.134191 18538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:36.134146 18541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:36.134162 18539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:36.134472 18533 server_base.cc:1034] running on GCE node
I20260618 23:35:36.134799 18533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:36.135178 18533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:36.136389 18533 hybrid_clock.cc:648] HybridClock initialized: now 1781825736136362 us; error 48 us; skew 500 ppm
Jun 18 23:35:36 dist-test-slave-6224 krb5kdc[18426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:36.139204 18533 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:35:36.139794 18533 webserver.cc:460] Webserver started at http://127.0.98.1:38645/ using document root <none> and password file <none>
I20260618 23:35:36.140049 18533 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:36.140101 18533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:36.140235 18533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:36.141361 18533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "9bc378b24b99401aae71b37d9bab1408"
format_stamp: "Formatted at 2026-06-18 23:35:36 on dist-test-slave-6224"
server_key: "8a8542341f67012c352326ee731a7695"
server_key_iv: "1f08d41e0193f8f837d0187e24b51f36"
server_key_version: "encryptionkey@0"
I20260618 23:35:36.141733 18533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9bc378b24b99401aae71b37d9bab1408"
format_stamp: "Formatted at 2026-06-18 23:35:36 on dist-test-slave-6224"
server_key: "8a8542341f67012c352326ee731a7695"
server_key_iv: "1f08d41e0193f8f837d0187e24b51f36"
server_key_version: "encryptionkey@0"
I20260618 23:35:36.143353 18533 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:35:36.144371 18548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:36.144654 18533 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:36.144776 18533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "9bc378b24b99401aae71b37d9bab1408"
format_stamp: "Formatted at 2026-06-18 23:35:36 on dist-test-slave-6224"
server_key: "8a8542341f67012c352326ee731a7695"
server_key_iv: "1f08d41e0193f8f837d0187e24b51f36"
server_key_version: "encryptionkey@0"
I20260618 23:35:36.144870 18533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:36.191362 18533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:36.193853 18533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:36.194024 18533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:36.194307 18533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:35:36.194687 18533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:35:36.194747 18533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:36.194792 18533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:35:36.194819 18533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:36.202136 18533 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39793
I20260618 23:35:36.202322 18661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39793 every 8 connection(s)
I20260618 23:35:36.202608 18533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:35:36.204767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 18533
I20260618 23:35:36.204905 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:35:36.205152 392 external_mini_cluster.cc:1428] Setting key a0af681e354d2b061f090cc459305cbf
Jun 18 23:35:36 dist-test-slave-6224 krb5kdc[18426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260618 23:35:36.215687 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:36.203660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48845 (local address 127.0.98.62:36837)
0618 23:35:36.203756 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:36.203760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:36.203966 (+ 206us) server_negotiation.cc:407] Connection header received
0618 23:35:36.205125 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:36.205127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:36.205142 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:36.205194 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:36.206265 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.207154 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.208478 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.208725 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.208850 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:36.211792 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:36.211795 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:36.211795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:36.211807 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:36.213728 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:36.214339 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:36.214341 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:36.214341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:36.214407 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:36.214786 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:36.214787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:36.214788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:36.214966 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:36.215045 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:36.215437 (+ 392us) server_negotiation.cc:299] Negotiation successful
0618 23:35:36.215534 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 23:35:36.216018 18664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:36.203511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36837 (local address 127.0.98.1:48845)
0618 23:35:36.203787 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:36.203850 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:36.205003 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:36.205288 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:36.205293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:36.205783 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:35:36.206126 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:36.206134 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.207300 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.207301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:36.208352 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:36.208355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.208856 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.208857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:36.209012 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:36.209701 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:35:36.209711 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:35:36.211682 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:35:36.213857 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:35:36.213864 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:35:36.213867 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:35:36.214204 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:35:36.214500 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:35:36.214501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:35:36.214502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:35:36.214669 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:35:36.215087 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:35:36.215094 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:35:36.215275 (+ 181us) client_negotiation.cc:769] Sending connection context
0618 23:35:36.215382 (+ 107us) client_negotiation.cc:240] Negotiation successful
0618 23:35:36.215539 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":150,"threads_started":1}
I20260618 23:35:36.216725 18662 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36837
I20260618 23:35:36.216922 18662 heartbeater.cc:464] Registering TS with master...
I20260618 23:35:36.217175 18662 heartbeater.cc:511] Master 127.0.98.62:36837 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260618 23:35:36.217871 18471 ts_manager.cc:194] Registered new tserver with Master: 9bc378b24b99401aae71b37d9bab1408 (127.0.98.1:39793)
I20260618 23:35:36.218554 18471 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260618 23:35:36.235847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36837
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:36.325881 18667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:36.326118 18667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:36.326144 18667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:36.327425 18667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:36.327476 18667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:36.327507 18667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:35:36.329387 18667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:36.329720 18667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:36.330073 18667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:36.336082 18672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:36.336089 18673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:36.336113 18675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:36.336150 18667 server_base.cc:1034] running on GCE node
I20260618 23:35:36.336558 18667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:36.336879 18667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:36.338074 18667 hybrid_clock.cc:648] HybridClock initialized: now 1781825736338054 us; error 43 us; skew 500 ppm
Jun 18 23:35:36 dist-test-slave-6224 krb5kdc[18426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:36.345229 18667 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:35:36.345759 18667 webserver.cc:460] Webserver started at http://127.0.98.2:40481/ using document root <none> and password file <none>
I20260618 23:35:36.346062 18667 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:36.346118 18667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:36.346256 18667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:36.347474 18667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "8a11c1f71acb42c794cae0e4a986f3bc"
format_stamp: "Formatted at 2026-06-18 23:35:36 on dist-test-slave-6224"
server_key: "1296178eed7f5ffa12813928baf91587"
server_key_iv: "db08c972f434cc9ddd124367a6eb191e"
server_key_version: "encryptionkey@0"
I20260618 23:35:36.348003 18667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8a11c1f71acb42c794cae0e4a986f3bc"
format_stamp: "Formatted at 2026-06-18 23:35:36 on dist-test-slave-6224"
server_key: "1296178eed7f5ffa12813928baf91587"
server_key_iv: "db08c972f434cc9ddd124367a6eb191e"
server_key_version: "encryptionkey@0"
I20260618 23:35:36.349730 18667 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:35:36.350775 18682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:36.351159 18667 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:36.351292 18667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "8a11c1f71acb42c794cae0e4a986f3bc"
format_stamp: "Formatted at 2026-06-18 23:35:36 on dist-test-slave-6224"
server_key: "1296178eed7f5ffa12813928baf91587"
server_key_iv: "db08c972f434cc9ddd124367a6eb191e"
server_key_version: "encryptionkey@0"
I20260618 23:35:36.351418 18667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:36.381091 18667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:36.383749 18667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:36.383934 18667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:36.384228 18667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:35:36.384599 18667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:35:36.384665 18667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:36.384704 18667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:35:36.384732 18667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:36.392009 18667 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45213
I20260618 23:35:36.392079 18795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45213 every 8 connection(s)
I20260618 23:35:36.392490 18667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
Jun 18 23:35:36 dist-test-slave-6224 krb5kdc[18426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:35:36.402285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 18667
I20260618 23:35:36.402474 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:35:36.402781 392 external_mini_cluster.cc:1428] Setting key 38bc3da4c75575d038ab130290d33fad
I20260618 23:35:36.403627 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:36.393512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60979 (local address 127.0.98.62:36837)
0618 23:35:36.393592 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:36.393594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:36.393880 (+ 286us) server_negotiation.cc:407] Connection header received
0618 23:35:36.394920 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:36.394922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:36.394935 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:36.394996 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:36.395768 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.396303 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.397388 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.397589 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.397684 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:36.400139 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:36.400141 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:36.400141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:36.400149 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:36.401554 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:36.402022 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:36.402023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:36.402024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:36.402081 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:36.402356 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:36.402357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:36.402357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:36.402552 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:36.402626 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:36.403345 (+ 719us) server_negotiation.cc:299] Negotiation successful
0618 23:35:36.403459 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 23:35:36.403923 18798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:36.393462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36837 (local address 127.0.98.2:60979)
0618 23:35:36.393712 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:36.393755 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:36.394792 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:36.395080 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:36.395083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:36.395366 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:35:36.395655 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:36.395660 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.396427 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.396428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:36.397272 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:36.397274 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.397694 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.397695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:36.397845 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:36.398517 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:35:36.398526 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:35:36.400020 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:35:36.401660 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:35:36.401667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:35:36.401669 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:35:36.401911 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:35:36.402150 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:35:36.402151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:35:36.402152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:35:36.402277 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:35:36.402666 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:35:36.402682 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:35:36.402953 (+ 271us) client_negotiation.cc:769] Sending connection context
0618 23:35:36.403298 (+ 345us) client_negotiation.cc:240] Negotiation successful
0618 23:35:36.403588 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":127,"threads_started":1}
I20260618 23:35:36.405010 18796 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36837
I20260618 23:35:36.405247 18796 heartbeater.cc:464] Registering TS with master...
I20260618 23:35:36.405534 18796 heartbeater.cc:511] Master 127.0.98.62:36837 requested a full tablet report, sending...
I20260618 23:35:36.406098 18470 ts_manager.cc:194] Registered new tserver with Master: 8a11c1f71acb42c794cae0e4a986f3bc (127.0.98.2:45213)
I20260618 23:35:36.406739 18470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60979
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260618 23:35:36.439121 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36837
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:36.528554 18801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:36.528784 18801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:36.528812 18801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:36.529910 18801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:36.529953 18801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:36.529979 18801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:35:36.531510 18801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:36.531791 18801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:36.532047 18801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:36.537987 18806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:36.538064 18807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:36.537981 18801 server_base.cc:1034] running on GCE node
W20260618 23:35:36.538034 18809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:36.538585 18801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:36.538921 18801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:36.540093 18801 hybrid_clock.cc:648] HybridClock initialized: now 1781825736540062 us; error 47 us; skew 500 ppm
Jun 18 23:35:36 dist-test-slave-6224 krb5kdc[18426](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:36.543177 18801 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:35:36.543780 18801 webserver.cc:460] Webserver started at http://127.0.98.3:41415/ using document root <none> and password file <none>
I20260618 23:35:36.544073 18801 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:36.544153 18801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:36.544322 18801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:36.545469 18801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "b1af98f5eff448bd8bb551aa53522c3f"
format_stamp: "Formatted at 2026-06-18 23:35:36 on dist-test-slave-6224"
server_key: "411ee2e3d24e0af06bb15b3e2101b2a8"
server_key_iv: "6abdc866951e9f3bcc688674976ef6a8"
server_key_version: "encryptionkey@0"
I20260618 23:35:36.545847 18801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b1af98f5eff448bd8bb551aa53522c3f"
format_stamp: "Formatted at 2026-06-18 23:35:36 on dist-test-slave-6224"
server_key: "411ee2e3d24e0af06bb15b3e2101b2a8"
server_key_iv: "6abdc866951e9f3bcc688674976ef6a8"
server_key_version: "encryptionkey@0"
I20260618 23:35:36.547397 18801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:35:36.548406 18816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:36.548736 18801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:35:36.548864 18801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "b1af98f5eff448bd8bb551aa53522c3f"
format_stamp: "Formatted at 2026-06-18 23:35:36 on dist-test-slave-6224"
server_key: "411ee2e3d24e0af06bb15b3e2101b2a8"
server_key_iv: "6abdc866951e9f3bcc688674976ef6a8"
server_key_version: "encryptionkey@0"
I20260618 23:35:36.548971 18801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:36.560513 18801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:36.562788 18801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:36.562968 18801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:36.563228 18801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:35:36.563629 18801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:35:36.563690 18801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:36.563728 18801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:35:36.563756 18801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:36.570968 18801 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42187
I20260618 23:35:36.571069 18929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42187 every 8 connection(s)
I20260618 23:35:36.571430 18801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:35:36.575815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 18801
I20260618 23:35:36.575943 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:35:36.576129 392 external_mini_cluster.cc:1428] Setting key 6b34c8c9f86420da419b71140b2b9882
Jun 18 23:35:36 dist-test-slave-6224 krb5kdc[18426](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825736, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:35:36.583052 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:36.572488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35429 (local address 127.0.98.62:36837)
0618 23:35:36.572577 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:36.572580 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:36.572839 (+ 259us) server_negotiation.cc:407] Connection header received
0618 23:35:36.573956 (+ 1117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:36.573958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:36.573971 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:36.574058 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:36.574838 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.575366 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.576856 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.577288 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.577370 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:36.580279 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:36.580281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:36.580281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:36.580289 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:36.581570 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:36.582027 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:36.582028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:36.582028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:36.582090 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:36.582350 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:36.582351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:36.582352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:36.582483 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:36.582563 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:36.582859 (+ 296us) server_negotiation.cc:299] Negotiation successful
0618 23:35:36.582941 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 23:35:36.583278 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:36.572437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36837 (local address 127.0.98.3:35429)
0618 23:35:36.572669 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:36.572716 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:36.573805 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:36.574149 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:36.574154 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:36.574480 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:35:36.574732 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:36.574736 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.575462 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.575463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:36.576734 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:36.576737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.577389 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.577390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:36.577548 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:36.578432 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:35:36.578443 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:35:36.580151 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:35:36.581668 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:35:36.581676 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:35:36.581678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:35:36.581912 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:35:36.582163 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:35:36.582163 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:35:36.582164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:35:36.582278 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:35:36.582554 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:35:36.582561 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:35:36.582720 (+ 159us) client_negotiation.cc:769] Sending connection context
0618 23:35:36.582839 (+ 119us) client_negotiation.cc:240] Negotiation successful
0618 23:35:36.583019 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":109,"threads_started":1}
I20260618 23:35:36.583954 18930 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36837
I20260618 23:35:36.584120 18930 heartbeater.cc:464] Registering TS with master...
I20260618 23:35:36.584375 18930 heartbeater.cc:511] Master 127.0.98.62:36837 requested a full tablet report, sending...
I20260618 23:35:36.584887 18470 ts_manager.cc:194] Registered new tserver with Master: b1af98f5eff448bd8bb551aa53522c3f (127.0.98.3:42187)
I20260618 23:35:36.585318 18470 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35429
I20260618 23:35:36.588660 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:35:36.597519 18510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:36.590099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47108 (local address 127.0.98.62:36837)
0618 23:35:36.590184 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:36.590187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:36.590329 (+ 142us) server_negotiation.cc:407] Connection header received
0618 23:35:36.590380 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:36.590381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:36.590392 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:36.590446 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:36.591022 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.591666 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.592424 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.592625 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.592701 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:36.594214 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:36.594215 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:36.594216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:36.594223 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:36.595936 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:36.596449 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:36.596450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:36.596450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:36.596514 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:36.596829 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:36.596830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:36.596830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:36.597003 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:36.597108 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:36.597284 (+ 176us) server_negotiation.cc:299] Negotiation successful
0618 23:35:36.597354 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 23:35:36.600054 18470 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:35:36.602073 18470 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:35:36.610347 18941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:36.605253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42187 (local address 127.0.0.1:43654)
0618 23:35:36.605409 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:36.605457 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:36.605538 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:36.606135 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:36.606140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:36.606143 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:36.606243 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:36.606246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.607690 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.607692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:36.608996 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:36.608999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.609945 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.609946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:36.610048 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:36.610063 (+ 15us) client_negotiation.cc:769] Sending connection context
0618 23:35:36.610141 (+ 78us) client_negotiation.cc:240] Negotiation successful
0618 23:35:36.610221 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":105,"thread_start_us":88,"threads_started":1}
I20260618 23:35:36.610467 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:36.605662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42616 (local address 127.0.98.1:39793)
0618 23:35:36.605852 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:36.605856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:36.606050 (+ 194us) server_negotiation.cc:407] Connection header received
0618 23:35:36.606132 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:36.606135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:36.606152 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:36.606232 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:36.606517 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.607610 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.609329 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.610052 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.610152 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:36.610209 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:36.610324 (+ 115us) server_negotiation.cc:299] Negotiation successful
0618 23:35:36.610365 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":87,"threads_started":1}
I20260618 23:35:36.610544 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:36.605925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44798 (local address 127.0.98.2:45213)
0618 23:35:36.606422 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:36.606426 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:36.606435 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:35:36.606470 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:36.606471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:36.606485 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:36.606582 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:36.606716 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.607457 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.609330 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.609915 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.609988 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:36.610058 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:36.610386 (+ 328us) server_negotiation.cc:299] Negotiation successful
0618 23:35:36.610441 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":115,"threads_started":1}
I20260618 23:35:36.610839 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:36.605515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39793 (local address 127.0.0.1:42616)
0618 23:35:36.605988 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:36.606000 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:36.606062 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:36.606321 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:36.606331 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:36.606333 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:36.606431 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:36.606433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.607730 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.607730 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:36.609218 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:36.609221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.610157 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.610158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:36.610242 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:36.610248 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:35:36.610309 (+ 61us) client_negotiation.cc:240] Negotiation successful
0618 23:35:36.610386 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"mutex_wait_us":55,"thread_start_us":105,"threads_started":1}
I20260618 23:35:36.611143 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:36.605391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43654 (local address 127.0.98.3:42187)
0618 23:35:36.605638 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:36.605644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:36.605660 (+ 16us) server_negotiation.cc:407] Connection header received
0618 23:35:36.605709 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:36.605712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:36.605731 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:36.605860 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:36.606324 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.607511 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.609098 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.609829 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.609916 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:36.609981 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:36.610950 (+ 969us) server_negotiation.cc:299] Negotiation successful
0618 23:35:36.611028 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":130,"threads_started":1}
I20260618 23:35:36.610347 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:36.605753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45213 (local address 127.0.0.1:44798)
0618 23:35:36.606156 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:36.606165 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:36.606208 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:36.606594 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:36.606595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:36.606598 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:36.606651 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:36.606653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.607593 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.607594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:36.609218 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:36.609221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.610091 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.610092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:36.610156 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:36.610159 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:35:36.610215 (+ 56us) client_negotiation.cc:240] Negotiation successful
0618 23:35:36.610255 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":88,"thread_start_us":88,"threads_started":1}
I20260618 23:35:36.611701 18596 tablet_service.cc:1467] Processing CreateTablet for tablet 32e38a6a2b7447c99b327b2be79a3536 (DEFAULT_TABLE table=test-table [id=7008ff1c17db4980a20c1a08594c230c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:35:36.611776 18864 tablet_service.cc:1467] Processing CreateTablet for tablet 32e38a6a2b7447c99b327b2be79a3536 (DEFAULT_TABLE table=test-table [id=7008ff1c17db4980a20c1a08594c230c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:35:36.612099 18864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32e38a6a2b7447c99b327b2be79a3536. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:36.612099 18596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32e38a6a2b7447c99b327b2be79a3536. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:36.612085 18730 tablet_service.cc:1467] Processing CreateTablet for tablet 32e38a6a2b7447c99b327b2be79a3536 (DEFAULT_TABLE table=test-table [id=7008ff1c17db4980a20c1a08594c230c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:35:36.612493 18730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32e38a6a2b7447c99b327b2be79a3536. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:36.615612 18947 tablet_bootstrap.cc:492] T 32e38a6a2b7447c99b327b2be79a3536 P 8a11c1f71acb42c794cae0e4a986f3bc: Bootstrap starting.
I20260618 23:35:36.615926 18949 tablet_bootstrap.cc:492] T 32e38a6a2b7447c99b327b2be79a3536 P b1af98f5eff448bd8bb551aa53522c3f: Bootstrap starting.
I20260618 23:35:36.616521 18947 tablet_bootstrap.cc:654] T 32e38a6a2b7447c99b327b2be79a3536 P 8a11c1f71acb42c794cae0e4a986f3bc: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:36.616607 18949 tablet_bootstrap.cc:654] T 32e38a6a2b7447c99b327b2be79a3536 P b1af98f5eff448bd8bb551aa53522c3f: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:36.617105 18949 log.cc:824] T 32e38a6a2b7447c99b327b2be79a3536 P b1af98f5eff448bd8bb551aa53522c3f: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:36.617108 18947 log.cc:824] T 32e38a6a2b7447c99b327b2be79a3536 P 8a11c1f71acb42c794cae0e4a986f3bc: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:36.617530 18948 tablet_bootstrap.cc:492] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408: Bootstrap starting.
I20260618 23:35:36.617951 18947 tablet_bootstrap.cc:492] T 32e38a6a2b7447c99b327b2be79a3536 P 8a11c1f71acb42c794cae0e4a986f3bc: No bootstrap required, opened a new log
I20260618 23:35:36.618014 18947 ts_tablet_manager.cc:1397] T 32e38a6a2b7447c99b327b2be79a3536 P 8a11c1f71acb42c794cae0e4a986f3bc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 23:35:36.618201 18948 tablet_bootstrap.cc:654] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:36.618696 18948 log.cc:824] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:36.619228 18949 tablet_bootstrap.cc:492] T 32e38a6a2b7447c99b327b2be79a3536 P b1af98f5eff448bd8bb551aa53522c3f: No bootstrap required, opened a new log
I20260618 23:35:36.619349 18949 ts_tablet_manager.cc:1397] T 32e38a6a2b7447c99b327b2be79a3536 P b1af98f5eff448bd8bb551aa53522c3f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 23:35:36.619941 18948 tablet_bootstrap.cc:492] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408: No bootstrap required, opened a new log
I20260618 23:35:36.620014 18948 ts_tablet_manager.cc:1397] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260618 23:35:36.620275 18947 raft_consensus.cc:348] T 32e38a6a2b7447c99b327b2be79a3536 P 8a11c1f71acb42c794cae0e4a986f3bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1af98f5eff448bd8bb551aa53522c3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42187 } } peers { permanent_uuid: "9bc378b24b99401aae71b37d9bab1408" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39793 } } peers { permanent_uuid: "8a11c1f71acb42c794cae0e4a986f3bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } }
I20260618 23:35:36.620467 18947 raft_consensus.cc:374] T 32e38a6a2b7447c99b327b2be79a3536 P 8a11c1f71acb42c794cae0e4a986f3bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:36.620512 18947 raft_consensus.cc:729] T 32e38a6a2b7447c99b327b2be79a3536 P 8a11c1f71acb42c794cae0e4a986f3bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a11c1f71acb42c794cae0e4a986f3bc, State: Initialized, Role: FOLLOWER
I20260618 23:35:36.620661 18947 consensus_queue.cc:260] T 32e38a6a2b7447c99b327b2be79a3536 P 8a11c1f71acb42c794cae0e4a986f3bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1af98f5eff448bd8bb551aa53522c3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42187 } } peers { permanent_uuid: "9bc378b24b99401aae71b37d9bab1408" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39793 } } peers { permanent_uuid: "8a11c1f71acb42c794cae0e4a986f3bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } }
I20260618 23:35:36.621023 18796 heartbeater.cc:503] Master 127.0.98.62:36837 was elected leader, sending a full tablet report...
I20260618 23:35:36.621034 18947 ts_tablet_manager.cc:1428] T 32e38a6a2b7447c99b327b2be79a3536 P 8a11c1f71acb42c794cae0e4a986f3bc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260618 23:35:36.621483 18949 raft_consensus.cc:348] T 32e38a6a2b7447c99b327b2be79a3536 P b1af98f5eff448bd8bb551aa53522c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1af98f5eff448bd8bb551aa53522c3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42187 } } peers { permanent_uuid: "9bc378b24b99401aae71b37d9bab1408" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39793 } } peers { permanent_uuid: "8a11c1f71acb42c794cae0e4a986f3bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } }
I20260618 23:35:36.621696 18949 raft_consensus.cc:374] T 32e38a6a2b7447c99b327b2be79a3536 P b1af98f5eff448bd8bb551aa53522c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:36.621738 18949 raft_consensus.cc:729] T 32e38a6a2b7447c99b327b2be79a3536 P b1af98f5eff448bd8bb551aa53522c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1af98f5eff448bd8bb551aa53522c3f, State: Initialized, Role: FOLLOWER
I20260618 23:35:36.621901 18949 consensus_queue.cc:260] T 32e38a6a2b7447c99b327b2be79a3536 P b1af98f5eff448bd8bb551aa53522c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1af98f5eff448bd8bb551aa53522c3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42187 } } peers { permanent_uuid: "9bc378b24b99401aae71b37d9bab1408" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39793 } } peers { permanent_uuid: "8a11c1f71acb42c794cae0e4a986f3bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } }
I20260618 23:35:36.622006 18948 raft_consensus.cc:348] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1af98f5eff448bd8bb551aa53522c3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42187 } } peers { permanent_uuid: "9bc378b24b99401aae71b37d9bab1408" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39793 } } peers { permanent_uuid: "8a11c1f71acb42c794cae0e4a986f3bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } }
I20260618 23:35:36.622188 18948 raft_consensus.cc:374] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:36.622229 18948 raft_consensus.cc:729] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bc378b24b99401aae71b37d9bab1408, State: Initialized, Role: FOLLOWER
I20260618 23:35:36.622329 18948 consensus_queue.cc:260] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1af98f5eff448bd8bb551aa53522c3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42187 } } peers { permanent_uuid: "9bc378b24b99401aae71b37d9bab1408" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39793 } } peers { permanent_uuid: "8a11c1f71acb42c794cae0e4a986f3bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } }
I20260618 23:35:36.622493 18949 ts_tablet_manager.cc:1428] T 32e38a6a2b7447c99b327b2be79a3536 P b1af98f5eff448bd8bb551aa53522c3f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:35:36.622537 18930 heartbeater.cc:503] Master 127.0.98.62:36837 was elected leader, sending a full tablet report...
I20260618 23:35:36.622607 18948 ts_tablet_manager.cc:1428] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:35:36.622671 18662 heartbeater.cc:503] Master 127.0.98.62:36837 was elected leader, sending a full tablet report...
W20260618 23:35:36.643236 18797 tablet.cc:2369] T 32e38a6a2b7447c99b327b2be79a3536 P 8a11c1f71acb42c794cae0e4a986f3bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:35:36.653544 18955 raft_consensus.cc:482] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:35:36.653748 18955 raft_consensus.cc:504] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1af98f5eff448bd8bb551aa53522c3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42187 } } peers { permanent_uuid: "9bc378b24b99401aae71b37d9bab1408" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39793 } } peers { permanent_uuid: "8a11c1f71acb42c794cae0e4a986f3bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } }
I20260618 23:35:36.654225 18955 leader_election.cc:288] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1af98f5eff448bd8bb551aa53522c3f (127.0.98.3:42187), 8a11c1f71acb42c794cae0e4a986f3bc (127.0.98.2:45213)
I20260618 23:35:36.658747 18664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:36.654548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42187 (local address 127.0.98.1:58179)
0618 23:35:36.654632 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:36.654652 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:36.654767 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:36.654982 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:36.654984 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:36.654986 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:36.655125 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:36.655129 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.656140 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.656140 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:36.657488 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:36.657492 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.658383 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.658384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:36.658490 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:36.658493 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:35:36.658558 (+ 65us) client_negotiation.cc:240] Negotiation successful
0618 23:35:36.658637 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":6}
I20260618 23:35:36.658746 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:36.654627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58179 (local address 127.0.98.3:42187)
0618 23:35:36.654688 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:36.654690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:36.654811 (+ 121us) server_negotiation.cc:407] Connection header received
0618 23:35:36.654847 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:36.654848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:36.654859 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:36.654921 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:36.655250 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.656038 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.657612 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.658242 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.658352 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:36.658361 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:36.658611 (+ 250us) server_negotiation.cc:299] Negotiation successful
0618 23:35:36.658662 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260618 23:35:36.658746 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:36.654920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59843 (local address 127.0.98.2:45213)
0618 23:35:36.655040 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:36.655043 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:36.655053 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:35:36.655080 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:36.655082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:36.655095 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:36.655145 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:36.655378 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.656131 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.657611 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.658245 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.658346 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:36.658359 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:36.658611 (+ 252us) server_negotiation.cc:299] Negotiation successful
0618 23:35:36.658662 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260618 23:35:36.658756 18956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:36.654686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45213 (local address 127.0.98.1:59843)
0618 23:35:36.654886 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:36.654896 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:36.654947 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:36.655198 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:36.655199 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:36.655201 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:36.655279 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:36.655281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.656219 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.656220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:36.657489 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:36.657492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.658383 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.658384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:36.658478 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:36.658484 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:35:36.658558 (+ 74us) client_negotiation.cc:240] Negotiation successful
0618 23:35:36.658635 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":52,"thread_start_us":97,"threads_started":1}
I20260618 23:35:36.659324 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e38a6a2b7447c99b327b2be79a3536" candidate_uuid: "9bc378b24b99401aae71b37d9bab1408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a11c1f71acb42c794cae0e4a986f3bc" is_pre_election: true
I20260618 23:35:36.659337 18884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e38a6a2b7447c99b327b2be79a3536" candidate_uuid: "9bc378b24b99401aae71b37d9bab1408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1af98f5eff448bd8bb551aa53522c3f" is_pre_election: true
I20260618 23:35:36.659490 18750 raft_consensus.cc:2454] T 32e38a6a2b7447c99b327b2be79a3536 P 8a11c1f71acb42c794cae0e4a986f3bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc378b24b99401aae71b37d9bab1408 in term 0.
I20260618 23:35:36.659533 18884 raft_consensus.cc:2454] T 32e38a6a2b7447c99b327b2be79a3536 P b1af98f5eff448bd8bb551aa53522c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9bc378b24b99401aae71b37d9bab1408 in term 0.
I20260618 23:35:36.659821 18550 leader_election.cc:302] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9bc378b24b99401aae71b37d9bab1408, b1af98f5eff448bd8bb551aa53522c3f; no voters:
I20260618 23:35:36.659960 18955 raft_consensus.cc:2781] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:35:36.660023 18955 raft_consensus.cc:482] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:35:36.660048 18955 raft_consensus.cc:3037] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:36.660934 18955 raft_consensus.cc:504] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1af98f5eff448bd8bb551aa53522c3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42187 } } peers { permanent_uuid: "9bc378b24b99401aae71b37d9bab1408" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39793 } } peers { permanent_uuid: "8a11c1f71acb42c794cae0e4a986f3bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } }
I20260618 23:35:36.661096 18955 leader_election.cc:288] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408 [CANDIDATE]: Term 1 election: Requested vote from peers b1af98f5eff448bd8bb551aa53522c3f (127.0.98.3:42187), 8a11c1f71acb42c794cae0e4a986f3bc (127.0.98.2:45213)
I20260618 23:35:36.661345 18884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e38a6a2b7447c99b327b2be79a3536" candidate_uuid: "9bc378b24b99401aae71b37d9bab1408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1af98f5eff448bd8bb551aa53522c3f"
I20260618 23:35:36.661417 18750 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32e38a6a2b7447c99b327b2be79a3536" candidate_uuid: "9bc378b24b99401aae71b37d9bab1408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a11c1f71acb42c794cae0e4a986f3bc"
I20260618 23:35:36.661492 18884 raft_consensus.cc:3037] T 32e38a6a2b7447c99b327b2be79a3536 P b1af98f5eff448bd8bb551aa53522c3f [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:36.661541 18750 raft_consensus.cc:3037] T 32e38a6a2b7447c99b327b2be79a3536 P 8a11c1f71acb42c794cae0e4a986f3bc [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:36.662365 18884 raft_consensus.cc:2454] T 32e38a6a2b7447c99b327b2be79a3536 P b1af98f5eff448bd8bb551aa53522c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc378b24b99401aae71b37d9bab1408 in term 1.
I20260618 23:35:36.662452 18750 raft_consensus.cc:2454] T 32e38a6a2b7447c99b327b2be79a3536 P 8a11c1f71acb42c794cae0e4a986f3bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9bc378b24b99401aae71b37d9bab1408 in term 1.
I20260618 23:35:36.662689 18550 leader_election.cc:302] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a11c1f71acb42c794cae0e4a986f3bc, 9bc378b24b99401aae71b37d9bab1408; no voters:
I20260618 23:35:36.662801 18955 raft_consensus.cc:2781] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:36.663004 18955 raft_consensus.cc:686] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408 [term 1 LEADER]: Becoming Leader. State: Replica: 9bc378b24b99401aae71b37d9bab1408, State: Running, Role: LEADER
I20260618 23:35:36.663139 18955 consensus_queue.cc:237] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1af98f5eff448bd8bb551aa53522c3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42187 } } peers { permanent_uuid: "9bc378b24b99401aae71b37d9bab1408" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39793 } } peers { permanent_uuid: "8a11c1f71acb42c794cae0e4a986f3bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } }
I20260618 23:35:36.664264 18470 catalog_manager.cc:5495] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9bc378b24b99401aae71b37d9bab1408 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9bc378b24b99401aae71b37d9bab1408" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1af98f5eff448bd8bb551aa53522c3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bc378b24b99401aae71b37d9bab1408" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a11c1f71acb42c794cae0e4a986f3bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 } health_report { overall_health: UNKNOWN } } }
I20260618 23:35:36.672333 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:36.669741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42628 (local address 127.0.98.1:39793)
0618 23:35:36.669821 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:36.669824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:36.669833 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:35:36.669877 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:36.669878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:36.669901 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:36.669983 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:35:36.670122 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.670610 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.671375 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.671629 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.671715 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:36.671949 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:35:36.672052 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:35:36.672167 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:36.672200 (+ 33us) server_negotiation.cc:299] Negotiation successful
0618 23:35:36.672211 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 23:35:36.674582 18750 raft_consensus.cc:1261] T 32e38a6a2b7447c99b327b2be79a3536 P 8a11c1f71acb42c794cae0e4a986f3bc [term 1 FOLLOWER]: Refusing update from remote peer 9bc378b24b99401aae71b37d9bab1408: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:35:36.674687 18884 raft_consensus.cc:1261] T 32e38a6a2b7447c99b327b2be79a3536 P b1af98f5eff448bd8bb551aa53522c3f [term 1 FOLLOWER]: Refusing update from remote peer 9bc378b24b99401aae71b37d9bab1408: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:35:36.675057 18955 consensus_queue.cc:1035] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a11c1f71acb42c794cae0e4a986f3bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:35:36.675272 18957 consensus_queue.cc:1035] T 32e38a6a2b7447c99b327b2be79a3536 P 9bc378b24b99401aae71b37d9bab1408 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1af98f5eff448bd8bb551aa53522c3f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:35:36.678304 18962 mvcc.cc:205] Tried to move back new op lower bound from 7298358217415819264 to 7298358217373065216. Current Snapshot: MvccSnapshot[applied={T|T < 7298358217415819264}]
I20260618 23:35:36.681550 18470 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:35:36.681689 18470 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:35:36.682723 18470 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57a980ed3df64239bb39ba52335fff30: Sending DeleteTablet for 3 replicas of tablet 32e38a6a2b7447c99b327b2be79a3536
I20260618 23:35:36.683194 18596 tablet_service.cc:1514] Processing DeleteTablet for tablet 32e38a6a2b7447c99b327b2be79a3536 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:35:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42616
I20260618 23:35:36.683238 18864 tablet_service.cc:1514] Processing DeleteTablet for tablet 32e38a6a2b7447c99b327b2be79a3536 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:35:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43654
I20260618 23:35:36.683279 18730 tablet_service.cc:1514] Processing DeleteTablet for tablet 32e38a6a2b7447c99b327b2be79a3536 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:35:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44798
I20260618 23:35:36.683516 18972 tablet_replica.cc:330] T 32e38a6a2b7447c99b327b2be79a3536 P 8a11c1f71acb42c794cae0e4a986f3bc: stopping tablet replica
I20260618 23:35:36.683660 18972 raft_consensus.cc:2229] T 32e38a6a2b7447c99b327b2be79a3536 P 8a11c1f71acb42c794cae0e4a986f3bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:35:36.683786 18972 raft_consensus.cc:2258] T 32e38a6a2b7447c99b327b2be79a3536 P 8a11c1f71acb42c794cae0e4a986f3bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:35:36.683880 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 18533
I20260618 23:35:36.685817 18971 tablet_replica.cc:330] T 32e38a6a2b7447c99b327b2be79a3536 P b1af98f5eff448bd8bb551aa53522c3f: stopping tablet replica
I20260618 23:35:36.685913 18971 raft_consensus.cc:2229] T 32e38a6a2b7447c99b327b2be79a3536 P b1af98f5eff448bd8bb551aa53522c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:35:36.686051 18971 raft_consensus.cc:2258] T 32e38a6a2b7447c99b327b2be79a3536 P b1af98f5eff448bd8bb551aa53522c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:35:36.686688 18971 ts_tablet_manager.cc:1905] T 32e38a6a2b7447c99b327b2be79a3536 P b1af98f5eff448bd8bb551aa53522c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:35:36.687968 18972 ts_tablet_manager.cc:1905] T 32e38a6a2b7447c99b327b2be79a3536 P 8a11c1f71acb42c794cae0e4a986f3bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:35:36.688931 18971 ts_tablet_manager.cc:1918] T 32e38a6a2b7447c99b327b2be79a3536 P b1af98f5eff448bd8bb551aa53522c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:35:36.689030 18971 log.cc:1196] T 32e38a6a2b7447c99b327b2be79a3536 P b1af98f5eff448bd8bb551aa53522c3f: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/32e38a6a2b7447c99b327b2be79a3536
I20260618 23:35:36.689347 18971 ts_tablet_manager.cc:1939] T 32e38a6a2b7447c99b327b2be79a3536 P b1af98f5eff448bd8bb551aa53522c3f: Deleting consensus metadata
I20260618 23:35:36.689939 18456 catalog_manager.cc:4841] TS b1af98f5eff448bd8bb551aa53522c3f (127.0.98.3:42187): tablet 32e38a6a2b7447c99b327b2be79a3536 (table test-table [id=7008ff1c17db4980a20c1a08594c230c]) successfully deleted
I20260618 23:35:36.690016 18972 ts_tablet_manager.cc:1918] T 32e38a6a2b7447c99b327b2be79a3536 P 8a11c1f71acb42c794cae0e4a986f3bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:35:36.690125 18972 log.cc:1196] T 32e38a6a2b7447c99b327b2be79a3536 P 8a11c1f71acb42c794cae0e4a986f3bc: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/32e38a6a2b7447c99b327b2be79a3536
I20260618 23:35:36.690464 18972 ts_tablet_manager.cc:1939] T 32e38a6a2b7447c99b327b2be79a3536 P 8a11c1f71acb42c794cae0e4a986f3bc: Deleting consensus metadata
W20260618 23:35:36.690652 18456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39793 (error 108)
I20260618 23:35:36.690834 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 18667
I20260618 23:35:36.690948 18456 catalog_manager.cc:4841] TS 8a11c1f71acb42c794cae0e4a986f3bc (127.0.98.2:45213): tablet 32e38a6a2b7447c99b327b2be79a3536 (table test-table [id=7008ff1c17db4980a20c1a08594c230c]) successfully deleted
I20260618 23:35:36.691994 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:36.691697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39793 (local address 127.0.0.1:42636)
0618 23:35:36.691863 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:36.691938 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260618 23:35:36.692641 18456 catalog_manager.cc:4584] TS 9bc378b24b99401aae71b37d9bab1408 (127.0.98.1:39793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 32e38a6a2b7447c99b327b2be79a3536: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39793: connect: Connection refused (error 111)
I20260618 23:35:36.697783 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 18801
I20260618 23:35:36.703392 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 18439
2026-06-18T23:35:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2081 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-18T23:35:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:35:36Z Disabled control of system clock
I20260618 23:35:36.735059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36795
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 23:35:36.829887 18977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:36.830210 18977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:36.830237 18977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:36.831892 18977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:35:36.831966 18977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:36.831980 18977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:36.831995 18977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:35:36.832008 18977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:35:36.833705 18977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36795
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:36.834003 18977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:36.834344 18977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:36.840761 18985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:36.840746 18982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:36.840746 18983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:36.841142 18977 server_base.cc:1034] running on GCE node
I20260618 23:35:36.841426 18977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:36.841737 18977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:36.842970 18977 hybrid_clock.cc:648] HybridClock initialized: now 1781825736842902 us; error 92 us; skew 500 ppm
I20260618 23:35:36.847653 18977 webserver.cc:460] Webserver started at http://127.0.98.62:37213/ using document root <none> and password file <none>
I20260618 23:35:36.847990 18977 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:36.848073 18977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:36.848239 18977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:36.849434 18977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "1d2da8e99d2a4a6381ad05b8acdacb95"
format_stamp: "Formatted at 2026-06-18 23:35:36 on dist-test-slave-6224"
server_key: "c3a336b9f2720a604887e29d6e555be6"
server_key_iv: "e61db79e33c405e5f2b62fab4a08b930"
server_key_version: "encryptionkey@0"
I20260618 23:35:36.849838 18977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "1d2da8e99d2a4a6381ad05b8acdacb95"
format_stamp: "Formatted at 2026-06-18 23:35:36 on dist-test-slave-6224"
server_key: "c3a336b9f2720a604887e29d6e555be6"
server_key_iv: "e61db79e33c405e5f2b62fab4a08b930"
server_key_version: "encryptionkey@0"
I20260618 23:35:36.851477 18977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:35:36.852711 18991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:36.853018 18977 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:36.853152 18977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "1d2da8e99d2a4a6381ad05b8acdacb95"
format_stamp: "Formatted at 2026-06-18 23:35:36 on dist-test-slave-6224"
server_key: "c3a336b9f2720a604887e29d6e555be6"
server_key_iv: "e61db79e33c405e5f2b62fab4a08b930"
server_key_version: "encryptionkey@0"
I20260618 23:35:36.853241 18977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:36.902822 18977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:36.903231 18977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:36.903465 18977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:36.908461 18977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36795
I20260618 23:35:36.908535 19043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36795 every 8 connection(s)
I20260618 23:35:36.909014 18977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:35:36.910297 19044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:36.912119 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 18977
I20260618 23:35:36.912283 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:35:36.912495 392 external_mini_cluster.cc:1428] Setting key e9891c93d858204a62adc8b7447f71cc
I20260618 23:35:36.913707 19044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95: Bootstrap starting.
I20260618 23:35:36.914476 19044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:36.914873 19044 log.cc:824] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:36.915781 19044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95: No bootstrap required, opened a new log
I20260618 23:35:36.918627 19044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2da8e99d2a4a6381ad05b8acdacb95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36795 } }
I20260618 23:35:36.918809 19044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:36.918835 19044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d2da8e99d2a4a6381ad05b8acdacb95, State: Initialized, Role: FOLLOWER
I20260618 23:35:36.918993 19044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2da8e99d2a4a6381ad05b8acdacb95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36795 } }
I20260618 23:35:36.919124 19044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:35:36.919174 19044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:35:36.919224 19044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:36.919629 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:36.913230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58684 (local address 127.0.98.62:36795)
0618 23:35:36.913464 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:36.913473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:36.913504 (+ 31us) server_negotiation.cc:407] Connection header received
0618 23:35:36.914213 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:36.914221 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:36.914291 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:36.914445 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:36.914814 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.915926 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.917156 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:36.917644 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:36.917795 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:36.918664 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:36.918668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:36.918669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:36.918923 (+ 254us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:36.918977 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:36.918989 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:36.919079 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:36.919211 (+ 132us) server_negotiation.cc:299] Negotiation successful
0618 23:35:36.919302 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":100,"threads_started":1}
I20260618 23:35:36.920147 19044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2da8e99d2a4a6381ad05b8acdacb95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36795 } }
I20260618 23:35:36.920272 19044 leader_election.cc:302] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d2da8e99d2a4a6381ad05b8acdacb95; no voters:
I20260618 23:35:36.920480 19044 leader_election.cc:288] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:35:36.920552 19049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:36.920780 19049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95 [term 1 LEADER]: Becoming Leader. State: Replica: 1d2da8e99d2a4a6381ad05b8acdacb95, State: Running, Role: LEADER
I20260618 23:35:36.920848 19044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:35:36.920943 19049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2da8e99d2a4a6381ad05b8acdacb95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36795 } }
I20260618 23:35:36.921442 19050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d2da8e99d2a4a6381ad05b8acdacb95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2da8e99d2a4a6381ad05b8acdacb95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36795 } } }
I20260618 23:35:36.921541 19050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:36.921639 19051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d2da8e99d2a4a6381ad05b8acdacb95. Latest consensus state: current_term: 1 leader_uuid: "1d2da8e99d2a4a6381ad05b8acdacb95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2da8e99d2a4a6381ad05b8acdacb95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36795 } } }
I20260618 23:35:36.921950 19058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:35:36.922024 19051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:36.922662 19058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:35:36.924742 19058 catalog_manager.cc:1347] Generated new cluster ID: 274559c9e6fd489596d62758dc32503a
I20260618 23:35:36.924795 19058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:35:36.938699 19058 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:35:36.939401 19058 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:35:36.947628 19058 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95: Generated new TSK 0
I20260618 23:35:36.947896 19058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:35:36.952976 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36795
--builtin_ntp_servers=127.0.98.20:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 23:35:37.048795 19068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:37.049104 19068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:37.049134 19068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:37.050367 19068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:37.050415 19068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:37.050443 19068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:35:37.052449 19068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:37.052789 19068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:37.053231 19068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:37.060056 19076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:37.060117 19074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:37.060056 19073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:37.060245 19068 server_base.cc:1034] running on GCE node
I20260618 23:35:37.060647 19068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:37.060993 19068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:37.062201 19068 hybrid_clock.cc:648] HybridClock initialized: now 1781825737062187 us; error 53 us; skew 500 ppm
I20260618 23:35:37.063905 19068 webserver.cc:460] Webserver started at http://127.0.98.1:40971/ using document root <none> and password file <none>
I20260618 23:35:37.064189 19068 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:37.064268 19068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:37.064438 19068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:37.065721 19068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "be3423b72689450eb67f2f5e9a5e414f"
format_stamp: "Formatted at 2026-06-18 23:35:37 on dist-test-slave-6224"
server_key: "a8042a8d98fb36afed4ed31d0b7e9813"
server_key_iv: "4c2fb5d954ba0ae3f0aa436a00a904ec"
server_key_version: "encryptionkey@0"
I20260618 23:35:37.066140 19068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "be3423b72689450eb67f2f5e9a5e414f"
format_stamp: "Formatted at 2026-06-18 23:35:37 on dist-test-slave-6224"
server_key: "a8042a8d98fb36afed4ed31d0b7e9813"
server_key_iv: "4c2fb5d954ba0ae3f0aa436a00a904ec"
server_key_version: "encryptionkey@0"
I20260618 23:35:37.067936 19068 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260618 23:35:37.069108 19082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:37.069501 19068 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:37.069664 19068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "be3423b72689450eb67f2f5e9a5e414f"
format_stamp: "Formatted at 2026-06-18 23:35:37 on dist-test-slave-6224"
server_key: "a8042a8d98fb36afed4ed31d0b7e9813"
server_key_iv: "4c2fb5d954ba0ae3f0aa436a00a904ec"
server_key_version: "encryptionkey@0"
I20260618 23:35:37.069777 19068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:37.086333 19068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:37.086869 19068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:37.087080 19068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:37.087410 19068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:35:37.087894 19068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:35:37.087956 19068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:37.087996 19068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:35:37.088045 19068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:37.096381 19068 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44493
I20260618 23:35:37.096498 19195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44493 every 8 connection(s)
I20260618 23:35:37.096911 19068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:35:37.100131 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 19068
I20260618 23:35:37.100241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:35:37.100498 392 external_mini_cluster.cc:1428] Setting key 822e00a7b2d11c85c764f9372154b239
I20260618 23:35:37.101766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36795
--builtin_ntp_servers=127.0.98.20:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 23:35:37.104693 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:37.097932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36179 (local address 127.0.98.62:36795)
0618 23:35:37.098030 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:37.098032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:37.098442 (+ 410us) server_negotiation.cc:407] Connection header received
0618 23:35:37.099090 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:37.099091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:37.099104 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:37.099192 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:37.100165 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.101013 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.102768 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.102977 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.103088 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:37.104340 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:37.104342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:37.104342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:37.104349 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:37.104368 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:37.104373 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:37.104447 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:37.104557 (+ 110us) server_negotiation.cc:299] Negotiation successful
0618 23:35:37.104594 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 23:35:37.105032 19198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:37.097915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36795 (local address 127.0.98.1:36179)
0618 23:35:37.098203 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:37.098269 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:37.098948 (+ 679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:37.099309 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:37.099313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:37.099323 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:35:37.100040 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:37.100046 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.101138 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.101139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:37.102637 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:37.102642 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.103108 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.103110 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:37.103313 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:37.104113 (+ 800us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:35:37.104113 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:35:37.104152 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:35:37.104185 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:35:37.104451 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:35:37.104458 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:35:37.104462 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:35:37.104563 (+ 101us) client_negotiation.cc:240] Negotiation successful
0618 23:35:37.104704 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":142,"threads_started":1}
I20260618 23:35:37.105813 19196 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36795
I20260618 23:35:37.105954 19196 heartbeater.cc:464] Registering TS with master...
I20260618 23:35:37.106180 19196 heartbeater.cc:511] Master 127.0.98.62:36795 requested a full tablet report, sending...
I20260618 23:35:37.106949 19008 ts_manager.cc:194] Registered new tserver with Master: be3423b72689450eb67f2f5e9a5e414f (127.0.98.1:44493)
I20260618 23:35:37.108049 19008 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:36179
W20260618 23:35:37.189800 19199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:37.190047 19199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:37.190071 19199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:37.191217 19199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:37.191262 19199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:37.191288 19199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:35:37.192664 19199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:37.192890 19199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:37.193174 19199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:37.199128 19204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:37.199184 19205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:37.199096 19207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:37.199584 19199 server_base.cc:1034] running on GCE node
I20260618 23:35:37.199838 19199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:37.200091 19199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:37.201285 19199 hybrid_clock.cc:648] HybridClock initialized: now 1781825737201268 us; error 42 us; skew 500 ppm
I20260618 23:35:37.202867 19199 webserver.cc:460] Webserver started at http://127.0.98.2:43613/ using document root <none> and password file <none>
I20260618 23:35:37.203202 19199 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:37.203284 19199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:37.203451 19199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:37.204648 19199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "0c9ecd5efa21407bb5064051720a6803"
format_stamp: "Formatted at 2026-06-18 23:35:37 on dist-test-slave-6224"
server_key: "2719b65d9c91bd742af9bfb6c0d09dc1"
server_key_iv: "5b7c1b2035b66ca65fb28944104a62c7"
server_key_version: "encryptionkey@0"
I20260618 23:35:37.205041 19199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0c9ecd5efa21407bb5064051720a6803"
format_stamp: "Formatted at 2026-06-18 23:35:37 on dist-test-slave-6224"
server_key: "2719b65d9c91bd742af9bfb6c0d09dc1"
server_key_iv: "5b7c1b2035b66ca65fb28944104a62c7"
server_key_version: "encryptionkey@0"
I20260618 23:35:37.206570 19199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:35:37.207683 19213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:37.207935 19199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:37.208056 19199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "0c9ecd5efa21407bb5064051720a6803"
format_stamp: "Formatted at 2026-06-18 23:35:37 on dist-test-slave-6224"
server_key: "2719b65d9c91bd742af9bfb6c0d09dc1"
server_key_iv: "5b7c1b2035b66ca65fb28944104a62c7"
server_key_version: "encryptionkey@0"
I20260618 23:35:37.208160 19199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:37.228250 19199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:37.228624 19199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:37.228801 19199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:37.229127 19199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:35:37.229547 19199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:35:37.229614 19199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:37.229647 19199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:35:37.229679 19199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:37.237143 19199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40785
I20260618 23:35:37.237191 19326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40785 every 8 connection(s)
I20260618 23:35:37.237609 19199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:35:37.243321 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:37.238453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37801 (local address 127.0.98.62:36795)
0618 23:35:37.238562 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:37.238565 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:37.238754 (+ 189us) server_negotiation.cc:407] Connection header received
0618 23:35:37.239285 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:37.239286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:37.239299 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:37.239394 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:37.240252 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.240837 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.241873 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.242091 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.242162 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:37.243043 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:37.243045 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:37.243045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:37.243051 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:37.243068 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:37.243072 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:37.243122 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:37.243220 (+ 98us) server_negotiation.cc:299] Negotiation successful
0618 23:35:37.243243 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260618 23:35:37.243441 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:37.238395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36795 (local address 127.0.98.2:37801)
0618 23:35:37.238599 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:37.238641 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:37.239151 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:37.239487 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:37.239490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:37.239499 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:35:37.240123 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:37.240128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.240935 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.240936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:37.241765 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:37.241767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.242185 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.242185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:37.242294 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:37.242892 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:35:37.242893 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:35:37.242921 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:35:37.242943 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:35:37.243136 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:35:37.243140 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:35:37.243142 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:35:37.243199 (+ 57us) client_negotiation.cc:240] Negotiation successful
0618 23:35:37.243278 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260618 23:35:37.244117 19327 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36795
I20260618 23:35:37.244303 19327 heartbeater.cc:464] Registering TS with master...
I20260618 23:35:37.244568 19327 heartbeater.cc:511] Master 127.0.98.62:36795 requested a full tablet report, sending...
I20260618 23:35:37.245162 19008 ts_manager.cc:194] Registered new tserver with Master: 0c9ecd5efa21407bb5064051720a6803 (127.0.98.2:40785)
I20260618 23:35:37.245602 19008 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:37801
I20260618 23:35:37.247795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 19199
I20260618 23:35:37.247898 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:35:37.248128 392 external_mini_cluster.cc:1428] Setting key 0d339c77b6bb975e00d3959ceafab7eb
I20260618 23:35:37.249248 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36795
--builtin_ntp_servers=127.0.98.20:46685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 23:35:37.342316 19330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:37.342653 19330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:37.342695 19330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:37.344029 19330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:37.344117 19330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:37.344167 19330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:35:37.345716 19330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:37.346066 19330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:37.346487 19330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:37.352762 19336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:37.352808 19338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:37.352814 19335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:37.353546 19330 server_base.cc:1034] running on GCE node
I20260618 23:35:37.353827 19330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:37.354123 19330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:37.355304 19330 hybrid_clock.cc:648] HybridClock initialized: now 1781825737355301 us; error 72 us; skew 500 ppm
I20260618 23:35:37.356879 19330 webserver.cc:460] Webserver started at http://127.0.98.3:46849/ using document root <none> and password file <none>
I20260618 23:35:37.357170 19330 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:37.357280 19330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:37.357443 19330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:37.358672 19330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "e184e6a685a64a8cbbc65ff7dfac831d"
format_stamp: "Formatted at 2026-06-18 23:35:37 on dist-test-slave-6224"
server_key: "f5781a50c840dde68a476f102d940955"
server_key_iv: "10a88021868a7bab50cf0dfb11918876"
server_key_version: "encryptionkey@0"
I20260618 23:35:37.359067 19330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e184e6a685a64a8cbbc65ff7dfac831d"
format_stamp: "Formatted at 2026-06-18 23:35:37 on dist-test-slave-6224"
server_key: "f5781a50c840dde68a476f102d940955"
server_key_iv: "10a88021868a7bab50cf0dfb11918876"
server_key_version: "encryptionkey@0"
I20260618 23:35:37.360778 19330 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260618 23:35:37.361791 19344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:37.362035 19330 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:37.362155 19330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "e184e6a685a64a8cbbc65ff7dfac831d"
format_stamp: "Formatted at 2026-06-18 23:35:37 on dist-test-slave-6224"
server_key: "f5781a50c840dde68a476f102d940955"
server_key_iv: "10a88021868a7bab50cf0dfb11918876"
server_key_version: "encryptionkey@0"
I20260618 23:35:37.362249 19330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:37.373451 19330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:37.373812 19330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:37.373980 19330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:37.374229 19330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:35:37.374593 19330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:35:37.374655 19330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:37.374699 19330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:35:37.374727 19330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:37.381719 19330 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35937
I20260618 23:35:37.381882 19457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35937 every 8 connection(s)
I20260618 23:35:37.382221 19330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:35:37.385277 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 19330
I20260618 23:35:37.385418 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:35:37.385609 392 external_mini_cluster.cc:1428] Setting key df52307ae26af7cca06d453a07be237f
I20260618 23:35:37.389107 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:37.383260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36795 (local address 127.0.98.3:35767)
0618 23:35:37.383487 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:37.383531 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:37.384111 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:37.384481 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:37.384485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:37.384493 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:35:37.385003 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:37.385007 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.386056 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.386057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:37.387290 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:37.387293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.387790 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.387791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:37.387922 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:37.388531 (+ 609us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:35:37.388531 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:35:37.388560 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:35:37.388583 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:35:37.388769 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:35:37.388771 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:35:37.388774 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:35:37.388828 (+ 54us) client_negotiation.cc:240] Negotiation successful
0618 23:35:37.388916 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":99,"threads_started":1}
I20260618 23:35:37.389135 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:37.383293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35767 (local address 127.0.98.62:36795)
0618 23:35:37.383391 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:37.383394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:37.383693 (+ 299us) server_negotiation.cc:407] Connection header received
0618 23:35:37.384261 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:37.384263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:37.384276 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:37.384369 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:37.385195 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.385923 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.387400 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.387699 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.387775 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:37.388685 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:37.388686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:37.388687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:37.388692 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:37.388710 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:37.388714 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:37.388769 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:37.388972 (+ 203us) server_negotiation.cc:299] Negotiation successful
0618 23:35:37.389012 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260618 23:35:37.390065 19458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36795
I20260618 23:35:37.390172 19458 heartbeater.cc:464] Registering TS with master...
I20260618 23:35:37.390331 19458 heartbeater.cc:511] Master 127.0.98.62:36795 requested a full tablet report, sending...
I20260618 23:35:37.390864 19008 ts_manager.cc:194] Registered new tserver with Master: e184e6a685a64a8cbbc65ff7dfac831d (127.0.98.3:35937)
I20260618 23:35:37.391266 19008 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:35767
I20260618 23:35:37.398478 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:35:37.403950 19047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:37.400369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.0.98.62:36795)
0618 23:35:37.400493 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:37.400496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:37.400627 (+ 131us) server_negotiation.cc:407] Connection header received
0618 23:35:37.400723 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:37.400724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:37.400735 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:37.400816 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:37.401023 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.401570 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.402380 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.402571 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.402652 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:37.403545 (+ 893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:37.403570 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:37.403570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:37.403575 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:37.403593 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:37.403602 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:37.403703 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:37.403836 (+ 133us) server_negotiation.cc:299] Negotiation successful
0618 23:35:37.403849 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260618 23:35:37.406051 19008 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:35:37.407944 19008 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:35:37.415994 19467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:37.410866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44493 (local address 127.0.0.1:60498)
0618 23:35:37.411315 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:37.411328 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:37.411390 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:37.411715 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:37.411719 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:37.411722 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:37.411813 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:37.411815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.413310 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.413311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:37.414548 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:37.414550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.415629 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.415630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:37.415745 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:37.415763 (+ 18us) client_negotiation.cc:769] Sending connection context
0618 23:35:37.415814 (+ 51us) client_negotiation.cc:240] Negotiation successful
0618 23:35:37.415890 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":3,"thread_start_us":72,"threads_started":1}
I20260618 23:35:37.416427 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:37.410988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60498 (local address 127.0.98.1:44493)
0618 23:35:37.411394 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:37.411400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:37.411414 (+ 14us) server_negotiation.cc:407] Connection header received
0618 23:35:37.411472 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:37.411474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:37.411493 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:37.411630 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:37.411937 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.413214 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.414669 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.415466 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.416153 (+ 687us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:37.416249 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:37.416275 (+ 26us) server_negotiation.cc:299] Negotiation successful
0618 23:35:37.416336 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":156,"threads_started":1}
I20260618 23:35:37.416657 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:37.411149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35937 (local address 127.0.0.1:51228)
0618 23:35:37.411497 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:37.411506 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:37.411597 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:37.412153 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:37.412154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:37.412157 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:37.412205 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:37.412208 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.413124 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.413125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:37.414542 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:37.414545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.415770 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.415771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:37.415868 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:37.415872 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:35:37.415919 (+ 47us) client_negotiation.cc:240] Negotiation successful
0618 23:35:37.415961 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":25,"thread_start_us":99,"threads_started":1}
I20260618 23:35:37.416662 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:37.410907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44436 (local address 127.0.98.2:40785)
0618 23:35:37.411569 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:37.411575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:37.411588 (+ 13us) server_negotiation.cc:407] Connection header received
0618 23:35:37.411638 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:37.411641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:37.411659 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:37.411991 (+ 332us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:37.412049 (+ 58us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.413215 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.415443 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.416280 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.416383 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:37.416491 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:37.416520 (+ 29us) server_negotiation.cc:299] Negotiation successful
0618 23:35:37.416566 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":90,"threads_started":1}
I20260618 23:35:37.417016 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:37.411256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51228 (local address 127.0.98.3:35937)
0618 23:35:37.411976 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:37.411980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:37.411989 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:35:37.412022 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:37.412024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:37.412036 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:37.412148 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:37.412263 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.413021 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.414681 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.415502 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.415636 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:37.415741 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:37.416039 (+ 298us) server_negotiation.cc:299] Negotiation successful
0618 23:35:37.416446 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":97,"threads_started":1}
I20260618 23:35:37.416666 19466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:37.410863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40785 (local address 127.0.0.1:44436)
0618 23:35:37.411089 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:37.411133 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:37.411218 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:37.411774 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:37.411775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:37.411778 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:37.411900 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:37.411904 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.414665 (+ 2761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.414666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:37.415330 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:37.415331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.416385 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.416386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:37.416453 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:37.416457 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:35:37.416503 (+ 46us) client_negotiation.cc:240] Negotiation successful
0618 23:35:37.416555 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":97,"threads_started":1}
I20260618 23:35:37.417207 19130 tablet_service.cc:1467] Processing CreateTablet for tablet 00fdc2b7857e477fad37bf824e5d613e (DEFAULT_TABLE table=test-table [id=8bffcc7ac88b4511beba478d5a831a23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:35:37.417650 19130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00fdc2b7857e477fad37bf824e5d613e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:37.417958 19261 tablet_service.cc:1467] Processing CreateTablet for tablet 00fdc2b7857e477fad37bf824e5d613e (DEFAULT_TABLE table=test-table [id=8bffcc7ac88b4511beba478d5a831a23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:35:37.418093 19392 tablet_service.cc:1467] Processing CreateTablet for tablet 00fdc2b7857e477fad37bf824e5d613e (DEFAULT_TABLE table=test-table [id=8bffcc7ac88b4511beba478d5a831a23]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:35:37.418268 19261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00fdc2b7857e477fad37bf824e5d613e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:37.418439 19392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00fdc2b7857e477fad37bf824e5d613e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:37.422049 19473 tablet_bootstrap.cc:492] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f: Bootstrap starting.
I20260618 23:35:37.422502 19474 tablet_bootstrap.cc:492] T 00fdc2b7857e477fad37bf824e5d613e P e184e6a685a64a8cbbc65ff7dfac831d: Bootstrap starting.
I20260618 23:35:37.422765 19472 tablet_bootstrap.cc:492] T 00fdc2b7857e477fad37bf824e5d613e P 0c9ecd5efa21407bb5064051720a6803: Bootstrap starting.
I20260618 23:35:37.423660 19474 tablet_bootstrap.cc:654] T 00fdc2b7857e477fad37bf824e5d613e P e184e6a685a64a8cbbc65ff7dfac831d: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:37.423683 19472 tablet_bootstrap.cc:654] T 00fdc2b7857e477fad37bf824e5d613e P 0c9ecd5efa21407bb5064051720a6803: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:37.424032 19472 log.cc:824] T 00fdc2b7857e477fad37bf824e5d613e P 0c9ecd5efa21407bb5064051720a6803: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:37.424032 19474 log.cc:824] T 00fdc2b7857e477fad37bf824e5d613e P e184e6a685a64a8cbbc65ff7dfac831d: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:37.424824 19473 tablet_bootstrap.cc:654] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:37.424965 19472 tablet_bootstrap.cc:492] T 00fdc2b7857e477fad37bf824e5d613e P 0c9ecd5efa21407bb5064051720a6803: No bootstrap required, opened a new log
I20260618 23:35:37.425060 19472 ts_tablet_manager.cc:1397] T 00fdc2b7857e477fad37bf824e5d613e P 0c9ecd5efa21407bb5064051720a6803: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260618 23:35:37.425218 19473 log.cc:824] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:37.425995 19473 tablet_bootstrap.cc:492] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f: No bootstrap required, opened a new log
I20260618 23:35:37.426096 19473 ts_tablet_manager.cc:1397] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260618 23:35:37.427188 19472 raft_consensus.cc:348] T 00fdc2b7857e477fad37bf824e5d613e P 0c9ecd5efa21407bb5064051720a6803 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9ecd5efa21407bb5064051720a6803" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40785 } } peers { permanent_uuid: "e184e6a685a64a8cbbc65ff7dfac831d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35937 } } peers { permanent_uuid: "be3423b72689450eb67f2f5e9a5e414f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44493 } }
I20260618 23:35:37.427383 19472 raft_consensus.cc:374] T 00fdc2b7857e477fad37bf824e5d613e P 0c9ecd5efa21407bb5064051720a6803 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:37.427410 19472 raft_consensus.cc:729] T 00fdc2b7857e477fad37bf824e5d613e P 0c9ecd5efa21407bb5064051720a6803 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c9ecd5efa21407bb5064051720a6803, State: Initialized, Role: FOLLOWER
I20260618 23:35:37.427891 19474 tablet_bootstrap.cc:492] T 00fdc2b7857e477fad37bf824e5d613e P e184e6a685a64a8cbbc65ff7dfac831d: No bootstrap required, opened a new log
I20260618 23:35:37.427956 19474 ts_tablet_manager.cc:1397] T 00fdc2b7857e477fad37bf824e5d613e P e184e6a685a64a8cbbc65ff7dfac831d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260618 23:35:37.428570 19473 raft_consensus.cc:348] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9ecd5efa21407bb5064051720a6803" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40785 } } peers { permanent_uuid: "e184e6a685a64a8cbbc65ff7dfac831d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35937 } } peers { permanent_uuid: "be3423b72689450eb67f2f5e9a5e414f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44493 } }
I20260618 23:35:37.428750 19473 raft_consensus.cc:374] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:37.428797 19473 raft_consensus.cc:729] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be3423b72689450eb67f2f5e9a5e414f, State: Initialized, Role: FOLLOWER
I20260618 23:35:37.428928 19473 consensus_queue.cc:260] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9ecd5efa21407bb5064051720a6803" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40785 } } peers { permanent_uuid: "e184e6a685a64a8cbbc65ff7dfac831d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35937 } } peers { permanent_uuid: "be3423b72689450eb67f2f5e9a5e414f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44493 } }
I20260618 23:35:37.429291 19473 ts_tablet_manager.cc:1428] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:35:37.429306 19196 heartbeater.cc:503] Master 127.0.98.62:36795 was elected leader, sending a full tablet report...
I20260618 23:35:37.430357 19474 raft_consensus.cc:348] T 00fdc2b7857e477fad37bf824e5d613e P e184e6a685a64a8cbbc65ff7dfac831d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9ecd5efa21407bb5064051720a6803" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40785 } } peers { permanent_uuid: "e184e6a685a64a8cbbc65ff7dfac831d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35937 } } peers { permanent_uuid: "be3423b72689450eb67f2f5e9a5e414f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44493 } }
I20260618 23:35:37.430528 19474 raft_consensus.cc:374] T 00fdc2b7857e477fad37bf824e5d613e P e184e6a685a64a8cbbc65ff7dfac831d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:37.430577 19474 raft_consensus.cc:729] T 00fdc2b7857e477fad37bf824e5d613e P e184e6a685a64a8cbbc65ff7dfac831d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e184e6a685a64a8cbbc65ff7dfac831d, State: Initialized, Role: FOLLOWER
I20260618 23:35:37.430703 19474 consensus_queue.cc:260] T 00fdc2b7857e477fad37bf824e5d613e P e184e6a685a64a8cbbc65ff7dfac831d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9ecd5efa21407bb5064051720a6803" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40785 } } peers { permanent_uuid: "e184e6a685a64a8cbbc65ff7dfac831d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35937 } } peers { permanent_uuid: "be3423b72689450eb67f2f5e9a5e414f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44493 } }
I20260618 23:35:37.431051 19474 ts_tablet_manager.cc:1428] T 00fdc2b7857e477fad37bf824e5d613e P e184e6a685a64a8cbbc65ff7dfac831d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260618 23:35:37.431159 19458 heartbeater.cc:503] Master 127.0.98.62:36795 was elected leader, sending a full tablet report...
I20260618 23:35:37.431309 19472 consensus_queue.cc:260] T 00fdc2b7857e477fad37bf824e5d613e P 0c9ecd5efa21407bb5064051720a6803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9ecd5efa21407bb5064051720a6803" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40785 } } peers { permanent_uuid: "e184e6a685a64a8cbbc65ff7dfac831d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35937 } } peers { permanent_uuid: "be3423b72689450eb67f2f5e9a5e414f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44493 } }
I20260618 23:35:37.431684 19472 ts_tablet_manager.cc:1428] T 00fdc2b7857e477fad37bf824e5d613e P 0c9ecd5efa21407bb5064051720a6803: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20260618 23:35:37.431773 19327 heartbeater.cc:503] Master 127.0.98.62:36795 was elected leader, sending a full tablet report...
W20260618 23:35:37.488371 19328 tablet.cc:2369] T 00fdc2b7857e477fad37bf824e5d613e P 0c9ecd5efa21407bb5064051720a6803: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:35:37.554651 19478 raft_consensus.cc:482] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:35:37.554888 19478 raft_consensus.cc:504] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9ecd5efa21407bb5064051720a6803" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40785 } } peers { permanent_uuid: "e184e6a685a64a8cbbc65ff7dfac831d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35937 } } peers { permanent_uuid: "be3423b72689450eb67f2f5e9a5e414f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44493 } }
I20260618 23:35:37.555349 19478 leader_election.cc:288] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c9ecd5efa21407bb5064051720a6803 (127.0.98.2:40785), e184e6a685a64a8cbbc65ff7dfac831d (127.0.98.3:35937)
I20260618 23:35:37.559084 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:37.555821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38095 (local address 127.0.98.2:40785)
0618 23:35:37.555904 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:37.555906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:37.555924 (+ 18us) server_negotiation.cc:407] Connection header received
0618 23:35:37.555961 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:37.555963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:37.555988 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:37.556098 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:37.556367 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.556900 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.558177 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.558634 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.558705 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:37.558716 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:37.558938 (+ 222us) server_negotiation.cc:299] Negotiation successful
0618 23:35:37.558987 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 23:35:37.559083 19198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:37.555718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40785 (local address 127.0.98.1:38095)
0618 23:35:37.555796 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:37.555811 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:37.555871 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:37.556122 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:37.556124 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:37.556125 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:37.556222 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:37.556224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.557010 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.557012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:37.558069 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:37.558072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.558736 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.558737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:37.558812 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:37.558816 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:35:37.558903 (+ 87us) client_negotiation.cc:240] Negotiation successful
0618 23:35:37.558953 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260618 23:35:37.559521 19281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fdc2b7857e477fad37bf824e5d613e" candidate_uuid: "be3423b72689450eb67f2f5e9a5e414f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c9ecd5efa21407bb5064051720a6803" is_pre_election: true
I20260618 23:35:37.559774 19281 raft_consensus.cc:2454] T 00fdc2b7857e477fad37bf824e5d613e P 0c9ecd5efa21407bb5064051720a6803 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be3423b72689450eb67f2f5e9a5e414f in term 0.
I20260618 23:35:37.560089 19084 leader_election.cc:302] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c9ecd5efa21407bb5064051720a6803, be3423b72689450eb67f2f5e9a5e414f; no voters:
I20260618 23:35:37.560170 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:37.556026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52579 (local address 127.0.98.3:35937)
0618 23:35:37.556083 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:37.556085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:37.556099 (+ 14us) server_negotiation.cc:407] Connection header received
0618 23:35:37.556186 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:37.556186 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:37.556195 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:37.556318 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:37.557286 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.557991 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.559098 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.559716 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.559790 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:37.559804 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:37.560002 (+ 198us) server_negotiation.cc:299] Negotiation successful
0618 23:35:37.560083 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260618 23:35:37.560096 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:37.555862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35937 (local address 127.0.98.1:52579)
0618 23:35:37.556050 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:37.556060 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:37.556111 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:37.557023 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:37.557023 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:37.557026 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:37.557133 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:37.557134 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.558102 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.558103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:37.558988 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:37.558991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.559845 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.559846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:37.559923 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:37.559928 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:35:37.559984 (+ 56us) client_negotiation.cc:240] Negotiation successful
0618 23:35:37.560033 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":144,"threads_started":1}
I20260618 23:35:37.560461 19478 raft_consensus.cc:2781] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:35:37.560539 19478 raft_consensus.cc:482] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:35:37.560577 19478 raft_consensus.cc:3037] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:37.560678 19412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fdc2b7857e477fad37bf824e5d613e" candidate_uuid: "be3423b72689450eb67f2f5e9a5e414f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e184e6a685a64a8cbbc65ff7dfac831d" is_pre_election: true
I20260618 23:35:37.560824 19412 raft_consensus.cc:2454] T 00fdc2b7857e477fad37bf824e5d613e P e184e6a685a64a8cbbc65ff7dfac831d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be3423b72689450eb67f2f5e9a5e414f in term 0.
I20260618 23:35:37.561573 19478 raft_consensus.cc:504] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9ecd5efa21407bb5064051720a6803" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40785 } } peers { permanent_uuid: "e184e6a685a64a8cbbc65ff7dfac831d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35937 } } peers { permanent_uuid: "be3423b72689450eb67f2f5e9a5e414f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44493 } }
I20260618 23:35:37.561779 19478 leader_election.cc:288] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f [CANDIDATE]: Term 1 election: Requested vote from peers 0c9ecd5efa21407bb5064051720a6803 (127.0.98.2:40785), e184e6a685a64a8cbbc65ff7dfac831d (127.0.98.3:35937)
I20260618 23:35:37.562006 19281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fdc2b7857e477fad37bf824e5d613e" candidate_uuid: "be3423b72689450eb67f2f5e9a5e414f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c9ecd5efa21407bb5064051720a6803"
I20260618 23:35:37.562063 19412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00fdc2b7857e477fad37bf824e5d613e" candidate_uuid: "be3423b72689450eb67f2f5e9a5e414f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e184e6a685a64a8cbbc65ff7dfac831d"
I20260618 23:35:37.562155 19281 raft_consensus.cc:3037] T 00fdc2b7857e477fad37bf824e5d613e P 0c9ecd5efa21407bb5064051720a6803 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:37.562196 19412 raft_consensus.cc:3037] T 00fdc2b7857e477fad37bf824e5d613e P e184e6a685a64a8cbbc65ff7dfac831d [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:37.563081 19412 raft_consensus.cc:2454] T 00fdc2b7857e477fad37bf824e5d613e P e184e6a685a64a8cbbc65ff7dfac831d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be3423b72689450eb67f2f5e9a5e414f in term 1.
I20260618 23:35:37.563213 19281 raft_consensus.cc:2454] T 00fdc2b7857e477fad37bf824e5d613e P 0c9ecd5efa21407bb5064051720a6803 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be3423b72689450eb67f2f5e9a5e414f in term 1.
I20260618 23:35:37.563303 19084 leader_election.cc:302] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be3423b72689450eb67f2f5e9a5e414f, e184e6a685a64a8cbbc65ff7dfac831d; no voters:
I20260618 23:35:37.563436 19478 raft_consensus.cc:2781] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:37.563716 19478 raft_consensus.cc:686] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f [term 1 LEADER]: Becoming Leader. State: Replica: be3423b72689450eb67f2f5e9a5e414f, State: Running, Role: LEADER
I20260618 23:35:37.563818 19478 consensus_queue.cc:237] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9ecd5efa21407bb5064051720a6803" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40785 } } peers { permanent_uuid: "e184e6a685a64a8cbbc65ff7dfac831d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35937 } } peers { permanent_uuid: "be3423b72689450eb67f2f5e9a5e414f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44493 } }
I20260618 23:35:37.564832 19007 catalog_manager.cc:5495] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f reported cstate change: term changed from 0 to 1, leader changed from <none> to be3423b72689450eb67f2f5e9a5e414f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "be3423b72689450eb67f2f5e9a5e414f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c9ecd5efa21407bb5064051720a6803" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e184e6a685a64a8cbbc65ff7dfac831d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be3423b72689450eb67f2f5e9a5e414f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44493 } health_report { overall_health: HEALTHY } } }
W20260618 23:35:37.597856 19197 tablet.cc:2369] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:35:37.601665 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:37.598837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60502 (local address 127.0.98.1:44493)
0618 23:35:37.598921 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:37.598923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:37.598934 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:35:37.598981 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:37.598982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:37.599006 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:37.599110 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:35:37.599260 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.599912 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.600808 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:37.601021 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:37.601115 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:37.601271 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:35:37.601383 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:35:37.601470 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:37.601542 (+ 72us) server_negotiation.cc:299] Negotiation successful
0618 23:35:37.601556 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 23:35:37.604146 19412 raft_consensus.cc:1261] T 00fdc2b7857e477fad37bf824e5d613e P e184e6a685a64a8cbbc65ff7dfac831d [term 1 FOLLOWER]: Refusing update from remote peer be3423b72689450eb67f2f5e9a5e414f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:35:37.604218 19281 raft_consensus.cc:1261] T 00fdc2b7857e477fad37bf824e5d613e P 0c9ecd5efa21407bb5064051720a6803 [term 1 FOLLOWER]: Refusing update from remote peer be3423b72689450eb67f2f5e9a5e414f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:35:37.604635 19482 consensus_queue.cc:1035] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e184e6a685a64a8cbbc65ff7dfac831d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:35:37.604811 19478 consensus_queue.cc:1035] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c9ecd5efa21407bb5064051720a6803" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:35:37.608408 19486 mvcc.cc:205] Tried to move back new op lower bound from 7298358221223264256 to 7298358221062164480. Current Snapshot: MvccSnapshot[applied={T|T < 7298358221223264256}]
I20260618 23:35:37.609189 19488 mvcc.cc:205] Tried to move back new op lower bound from 7298358221223264256 to 7298358221062164480. Current Snapshot: MvccSnapshot[applied={T|T < 7298358221223264256}]
I20260618 23:35:37.612097 19007 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:35:37.612301 19007 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:35:37.613327 19007 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1d2da8e99d2a4a6381ad05b8acdacb95: Sending DeleteTablet for 3 replicas of tablet 00fdc2b7857e477fad37bf824e5d613e
I20260618 23:35:37.613816 19130 tablet_service.cc:1514] Processing DeleteTablet for tablet 00fdc2b7857e477fad37bf824e5d613e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:35:37 UTC) from {username='slave'} at 127.0.0.1:60498
I20260618 23:35:37.613865 19261 tablet_service.cc:1514] Processing DeleteTablet for tablet 00fdc2b7857e477fad37bf824e5d613e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:35:37 UTC) from {username='slave'} at 127.0.0.1:44436
I20260618 23:35:37.613844 19392 tablet_service.cc:1514] Processing DeleteTablet for tablet 00fdc2b7857e477fad37bf824e5d613e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:35:37 UTC) from {username='slave'} at 127.0.0.1:51228
I20260618 23:35:37.614223 19495 tablet_replica.cc:330] T 00fdc2b7857e477fad37bf824e5d613e P 0c9ecd5efa21407bb5064051720a6803: stopping tablet replica
I20260618 23:35:37.614342 19496 tablet_replica.cc:330] T 00fdc2b7857e477fad37bf824e5d613e P e184e6a685a64a8cbbc65ff7dfac831d: stopping tablet replica
I20260618 23:35:37.614470 19494 tablet_replica.cc:330] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f: stopping tablet replica
I20260618 23:35:37.614581 19494 raft_consensus.cc:2229] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:35:37.614594 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 19068
I20260618 23:35:37.614640 19495 raft_consensus.cc:2229] T 00fdc2b7857e477fad37bf824e5d613e P 0c9ecd5efa21407bb5064051720a6803 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:35:37.614496 19496 raft_consensus.cc:2229] T 00fdc2b7857e477fad37bf824e5d613e P e184e6a685a64a8cbbc65ff7dfac831d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:35:37.614722 19494 raft_consensus.cc:2258] T 00fdc2b7857e477fad37bf824e5d613e P be3423b72689450eb67f2f5e9a5e414f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:35:37.614863 19495 raft_consensus.cc:2258] T 00fdc2b7857e477fad37bf824e5d613e P 0c9ecd5efa21407bb5064051720a6803 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:35:37.615507 19496 raft_consensus.cc:2258] T 00fdc2b7857e477fad37bf824e5d613e P e184e6a685a64a8cbbc65ff7dfac831d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:35:37.616235 19495 ts_tablet_manager.cc:1905] T 00fdc2b7857e477fad37bf824e5d613e P 0c9ecd5efa21407bb5064051720a6803: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:35:37.616235 19496 ts_tablet_manager.cc:1905] T 00fdc2b7857e477fad37bf824e5d613e P e184e6a685a64a8cbbc65ff7dfac831d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:35:37.619060 19496 ts_tablet_manager.cc:1918] T 00fdc2b7857e477fad37bf824e5d613e P e184e6a685a64a8cbbc65ff7dfac831d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:35:37.619092 19495 ts_tablet_manager.cc:1918] T 00fdc2b7857e477fad37bf824e5d613e P 0c9ecd5efa21407bb5064051720a6803: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:35:37.619175 19495 log.cc:1196] T 00fdc2b7857e477fad37bf824e5d613e P 0c9ecd5efa21407bb5064051720a6803: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/00fdc2b7857e477fad37bf824e5d613e
I20260618 23:35:37.619341 19496 log.cc:1196] T 00fdc2b7857e477fad37bf824e5d613e P e184e6a685a64a8cbbc65ff7dfac831d: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/00fdc2b7857e477fad37bf824e5d613e
I20260618 23:35:37.619465 19495 ts_tablet_manager.cc:1939] T 00fdc2b7857e477fad37bf824e5d613e P 0c9ecd5efa21407bb5064051720a6803: Deleting consensus metadata
I20260618 23:35:37.619773 19496 ts_tablet_manager.cc:1939] T 00fdc2b7857e477fad37bf824e5d613e P e184e6a685a64a8cbbc65ff7dfac831d: Deleting consensus metadata
I20260618 23:35:37.620142 18993 catalog_manager.cc:4841] TS 0c9ecd5efa21407bb5064051720a6803 (127.0.98.2:40785): tablet 00fdc2b7857e477fad37bf824e5d613e (table test-table [id=8bffcc7ac88b4511beba478d5a831a23]) successfully deleted
I20260618 23:35:37.620385 18993 catalog_manager.cc:4841] TS e184e6a685a64a8cbbc65ff7dfac831d (127.0.98.3:35937): tablet 00fdc2b7857e477fad37bf824e5d613e (table test-table [id=8bffcc7ac88b4511beba478d5a831a23]) successfully deleted
W20260618 23:35:37.621318 18992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44493 (error 108)
I20260618 23:35:37.622071 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:37.621868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44493 (local address 127.0.0.1:60504)
0618 23:35:37.621947 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:37.622031 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260618 23:35:37.622146 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 19199
W20260618 23:35:37.622228 18992 catalog_manager.cc:4584] TS be3423b72689450eb67f2f5e9a5e414f (127.0.98.1:44493): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 00fdc2b7857e477fad37bf824e5d613e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44493: connect: Connection refused (error 111)
I20260618 23:35:37.627941 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 19330
I20260618 23:35:37.633517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 18977
2026-06-18T23:35:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (929 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:35:37 dist-test-slave-6224 krb5kdc[19500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:35:37 dist-test-slave-6224 krb5kdc[19500](info): set up 2 sockets
Jun 18 23:35:37 dist-test-slave-6224 krb5kdc[19500](info): commencing operation
krb5kdc: starting...
W20260618 23:35:38.707949 392 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:35:38 dist-test-slave-6224 krb5kdc[19500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:35:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:35:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:35:38.843586 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40279
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:38.942014 19513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:38.942350 19513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:38.942396 19513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:38.943801 19513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:35:38.943892 19513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:38.943917 19513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:38.943939 19513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:35:38.943969 19513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:35:38.946019 19513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:38.946419 19513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:38.946714 19513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:38.953011 19518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:38.953001 19519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:38.953228 19513 server_base.cc:1034] running on GCE node
W20260618 23:35:38.953001 19521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:38.953954 19513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:38.954382 19513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:38.955642 19513 hybrid_clock.cc:648] HybridClock initialized: now 1781825738955603 us; error 54 us; skew 500 ppm
Jun 18 23:35:38 dist-test-slave-6224 krb5kdc[19500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:38.958479 19513 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:35:38.958966 19513 webserver.cc:460] Webserver started at http://127.0.98.62:36589/ using document root <none> and password file <none>
I20260618 23:35:38.959252 19513 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:38.959332 19513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:38.959496 19513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:38.960695 19513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "3eed096eca414dbd9f7b553ba4c1fa43"
format_stamp: "Formatted at 2026-06-18 23:35:38 on dist-test-slave-6224"
server_key: "f8636e04a392eddd9841abbf07e7d77e"
server_key_iv: "de0df7a07a32de9428cb95a64d1bc57c"
server_key_version: "encryptionkey@0"
I20260618 23:35:38.961092 19513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "3eed096eca414dbd9f7b553ba4c1fa43"
format_stamp: "Formatted at 2026-06-18 23:35:38 on dist-test-slave-6224"
server_key: "f8636e04a392eddd9841abbf07e7d77e"
server_key_iv: "de0df7a07a32de9428cb95a64d1bc57c"
server_key_version: "encryptionkey@0"
I20260618 23:35:38.962632 19513 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:35:38.963740 19528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:38.964061 19513 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:38.964160 19513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "3eed096eca414dbd9f7b553ba4c1fa43"
format_stamp: "Formatted at 2026-06-18 23:35:38 on dist-test-slave-6224"
server_key: "f8636e04a392eddd9841abbf07e7d77e"
server_key_iv: "de0df7a07a32de9428cb95a64d1bc57c"
server_key_version: "encryptionkey@0"
I20260618 23:35:38.964291 19513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:38.980186 19513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:38.982753 19513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:38.982874 19513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:38.987994 19513 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40279
I20260618 23:35:38.988072 19580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40279 every 8 connection(s)
I20260618 23:35:38.988487 19513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:35:38.989553 19581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:38.989924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 19513
I20260618 23:35:38.990069 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:35:38.990306 392 external_mini_cluster.cc:1428] Setting key d249442e89b8c7f7b26b81952dcdfd54
I20260618 23:35:38.992895 19581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43: Bootstrap starting.
I20260618 23:35:38.993595 19581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:38.993939 19581 log.cc:824] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:38.994743 19581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43: No bootstrap required, opened a new log
I20260618 23:35:38.997128 19581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eed096eca414dbd9f7b553ba4c1fa43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40279 } }
I20260618 23:35:38.997339 19581 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:38.997394 19581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eed096eca414dbd9f7b553ba4c1fa43, State: Initialized, Role: FOLLOWER
Jun 18 23:35:38 dist-test-slave-6224 krb5kdc[19500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:35:38.997570 19581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eed096eca414dbd9f7b553ba4c1fa43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40279 } }
I20260618 23:35:38.997707 19581 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:35:38.997743 19581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:35:38.997813 19581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:38.998685 19581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eed096eca414dbd9f7b553ba4c1fa43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40279 } }
I20260618 23:35:38.998817 19581 leader_election.cc:302] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3eed096eca414dbd9f7b553ba4c1fa43; no voters:
I20260618 23:35:38.999054 19581 leader_election.cc:288] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:35:38.999094 19586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:38.999235 19586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43 [term 1 LEADER]: Becoming Leader. State: Replica: 3eed096eca414dbd9f7b553ba4c1fa43, State: Running, Role: LEADER
I20260618 23:35:38.999346 19586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eed096eca414dbd9f7b553ba4c1fa43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40279 } }
I20260618 23:35:38.999394 19581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:35:39.000134 19587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3eed096eca414dbd9f7b553ba4c1fa43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eed096eca414dbd9f7b553ba4c1fa43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40279 } } }
I20260618 23:35:39.000666 19587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:39.000558 19588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3eed096eca414dbd9f7b553ba4c1fa43. Latest consensus state: current_term: 1 leader_uuid: "3eed096eca414dbd9f7b553ba4c1fa43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3eed096eca414dbd9f7b553ba4c1fa43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40279 } } }
I20260618 23:35:39.000989 19588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43 [sys.catalog]: This master's current role is: LEADER
W20260618 23:35:39.002050 19602 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:35:39.002133 19602 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:35:39.002197 19596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:35:39.002377 19596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:35:39.002933 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:38.990961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48120 (local address 127.0.98.62:40279)
0618 23:35:38.991239 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:38.991248 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:38.991280 (+ 32us) server_negotiation.cc:407] Connection header received
0618 23:35:38.991943 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:38.991951 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:38.992005 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:38.992167 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:38.992877 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:38.993952 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:38.994746 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:38.995260 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:38.995411 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:38.998346 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:38.998351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:38.998351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:38.998363 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:39.000798 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:39.001380 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:39.001383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:39.001383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:39.001475 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:39.001784 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:39.001785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:39.001785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:39.002131 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:39.002278 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:39.002522 (+ 244us) server_negotiation.cc:299] Negotiation successful
0618 23:35:39.002682 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":112,"threads_started":1}
I20260618 23:35:39.004701 19596 catalog_manager.cc:1347] Generated new cluster ID: 75d63983a8ff4ee590ae43e564c00bff
I20260618 23:35:39.004783 19596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:35:39.020769 19596 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:35:39.021534 19596 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:35:39.025120 19596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43: Generated new TSK 0
I20260618 23:35:39.025363 19596 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260618 23:35:39.060389 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40279
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:39.160534 19607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:39.160856 19607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:39.160910 19607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:39.162338 19607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:39.162420 19607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:39.162475 19607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:35:39.164644 19607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:39.164994 19607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:39.165277 19607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:39.171655 19613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:39.171734 19612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:39.171836 19607 server_base.cc:1034] running on GCE node
W20260618 23:35:39.171675 19615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:39.172323 19607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:39.172670 19607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:39.173890 19607 hybrid_clock.cc:648] HybridClock initialized: now 1781825739173857 us; error 55 us; skew 500 ppm
Jun 18 23:35:39 dist-test-slave-6224 krb5kdc[19500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:39.177052 19607 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260618 23:35:39.177682 19607 webserver.cc:460] Webserver started at http://127.0.98.1:35871/ using document root <none> and password file <none>
I20260618 23:35:39.177984 19607 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:39.178061 19607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:39.178227 19607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:39.179481 19607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "985ec866adfb43128d95efafe2293e69"
format_stamp: "Formatted at 2026-06-18 23:35:39 on dist-test-slave-6224"
server_key: "56d9833537ce5b567ae69d5556538956"
server_key_iv: "749994a01d91e327ce192de8f7f7ef93"
server_key_version: "encryptionkey@0"
I20260618 23:35:39.179978 19607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "985ec866adfb43128d95efafe2293e69"
format_stamp: "Formatted at 2026-06-18 23:35:39 on dist-test-slave-6224"
server_key: "56d9833537ce5b567ae69d5556538956"
server_key_iv: "749994a01d91e327ce192de8f7f7ef93"
server_key_version: "encryptionkey@0"
I20260618 23:35:39.181550 19607 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260618 23:35:39.182493 19622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:39.182797 19607 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:39.182911 19607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "985ec866adfb43128d95efafe2293e69"
format_stamp: "Formatted at 2026-06-18 23:35:39 on dist-test-slave-6224"
server_key: "56d9833537ce5b567ae69d5556538956"
server_key_iv: "749994a01d91e327ce192de8f7f7ef93"
server_key_version: "encryptionkey@0"
I20260618 23:35:39.183002 19607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:39.201484 19607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:39.204103 19607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:39.204321 19607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:39.204638 19607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:35:39.205036 19607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:35:39.205104 19607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:39.205161 19607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:35:39.205196 19607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:39.213449 19607 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46453
I20260618 23:35:39.213501 19735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46453 every 8 connection(s)
I20260618 23:35:39.213920 19607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:35:39.217931 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 19607
I20260618 23:35:39.218089 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:35:39.218314 392 external_mini_cluster.cc:1428] Setting key 7cf3a91f1de4717c50ccb77f7c79a37c
Jun 18 23:35:39 dist-test-slave-6224 krb5kdc[19500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:35:39.226501 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.214909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35563 (local address 127.0.98.62:40279)
0618 23:35:39.215003 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:39.215005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:39.215242 (+ 237us) server_negotiation.cc:407] Connection header received
0618 23:35:39.216310 (+ 1068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:39.216312 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:39.216325 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:39.216414 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:39.217134 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.217735 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.219504 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.219824 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.219932 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:39.223125 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:39.223126 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:39.223126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:39.223134 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:39.224879 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:39.225363 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:39.225364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:39.225365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:39.225434 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:39.225727 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:39.225728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:39.225728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:39.225860 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:39.225965 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:39.226278 (+ 313us) server_negotiation.cc:299] Negotiation successful
0618 23:35:39.226394 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 23:35:39.226728 19738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.214799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40279 (local address 127.0.98.1:35563)
0618 23:35:39.215054 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:39.215099 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:39.216172 (+ 1073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:39.216499 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:39.216503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:39.216814 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:35:39.217027 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:39.217031 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.217891 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.217894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:39.219396 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:39.219400 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.219946 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.219947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:39.220107 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:39.220775 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:35:39.220790 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:35:39.222935 (+ 2145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:35:39.225000 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:35:39.225008 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:35:39.225011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:35:39.225261 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:35:39.225511 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:35:39.225511 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:35:39.225512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:35:39.225649 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:35:39.225956 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:35:39.225964 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:35:39.226132 (+ 168us) client_negotiation.cc:769] Sending connection context
0618 23:35:39.226258 (+ 126us) client_negotiation.cc:240] Negotiation successful
0618 23:35:39.226442 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":141,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260618 23:35:39.227352 19736 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40279
I20260618 23:35:39.227504 19736 heartbeater.cc:464] Registering TS with master...
I20260618 23:35:39.227751 19736 heartbeater.cc:511] Master 127.0.98.62:40279 requested a full tablet report, sending...
I20260618 23:35:39.228461 19545 ts_manager.cc:194] Registered new tserver with Master: 985ec866adfb43128d95efafe2293e69 (127.0.98.1:46453)
I20260618 23:35:39.229167 19545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260618 23:35:39.247015 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40279
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:39.332696 19741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:39.332914 19741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:39.332938 19741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:39.334012 19741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:39.334051 19741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:39.334077 19741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:35:39.335479 19741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:39.335690 19741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:39.335876 19741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:39.341226 19747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:39.341240 19746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:39.341306 19749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:39.341627 19741 server_base.cc:1034] running on GCE node
I20260618 23:35:39.341848 19741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:39.342196 19741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:39.343420 19741 hybrid_clock.cc:648] HybridClock initialized: now 1781825739343405 us; error 28 us; skew 500 ppm
Jun 18 23:35:39 dist-test-slave-6224 krb5kdc[19500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:39.346427 19741 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260618 23:35:39.346982 19741 webserver.cc:460] Webserver started at http://127.0.98.2:39127/ using document root <none> and password file <none>
I20260618 23:35:39.347270 19741 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:39.347352 19741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:39.347535 19741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:39.348821 19741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "ff62710620ff406a8eabc2a41e37ad33"
format_stamp: "Formatted at 2026-06-18 23:35:39 on dist-test-slave-6224"
server_key: "17e03a252794fddf0de29866c2b4f189"
server_key_iv: "be56dc38d325480f753b3d67f835fb7f"
server_key_version: "encryptionkey@0"
I20260618 23:35:39.349258 19741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ff62710620ff406a8eabc2a41e37ad33"
format_stamp: "Formatted at 2026-06-18 23:35:39 on dist-test-slave-6224"
server_key: "17e03a252794fddf0de29866c2b4f189"
server_key_iv: "be56dc38d325480f753b3d67f835fb7f"
server_key_version: "encryptionkey@0"
I20260618 23:35:39.350813 19741 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:35:39.351915 19756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:39.352257 19741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:39.352357 19741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "ff62710620ff406a8eabc2a41e37ad33"
format_stamp: "Formatted at 2026-06-18 23:35:39 on dist-test-slave-6224"
server_key: "17e03a252794fddf0de29866c2b4f189"
server_key_iv: "be56dc38d325480f753b3d67f835fb7f"
server_key_version: "encryptionkey@0"
I20260618 23:35:39.352491 19741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:39.368237 19741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:39.370698 19741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:39.370889 19741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:39.371181 19741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:35:39.371579 19741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:35:39.371654 19741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:39.371699 19741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:35:39.371729 19741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:39.378540 19741 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43531
I20260618 23:35:39.378648 19869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43531 every 8 connection(s)
I20260618 23:35:39.378989 19741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:35:39.382953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 19741
I20260618 23:35:39.383083 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:35:39.383363 392 external_mini_cluster.cc:1428] Setting key 3dca100f0dbed7f527c8b24ce89edba3
Jun 18 23:35:39 dist-test-slave-6224 krb5kdc[19500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:35:39.390142 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.379955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36761 (local address 127.0.98.62:40279)
0618 23:35:39.380010 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:39.380013 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:39.380280 (+ 267us) server_negotiation.cc:407] Connection header received
0618 23:35:39.381363 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:39.381365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:39.381376 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:39.381462 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:39.382246 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.382789 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.384378 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.384598 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.384697 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:39.387108 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:39.387110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:39.387110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:39.387118 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:39.388693 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:39.389139 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:39.389140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:39.389141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:39.389202 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:39.389466 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:39.389467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:39.389467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:39.389590 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:39.389673 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:39.389950 (+ 277us) server_negotiation.cc:299] Negotiation successful
0618 23:35:39.390044 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260618 23:35:39.390337 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.379901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40279 (local address 127.0.98.2:36761)
0618 23:35:39.380109 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:39.380160 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:39.381214 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:39.381559 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:39.381563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:39.381885 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:35:39.382114 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:39.382118 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.382951 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.382953 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:39.384268 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:39.384276 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.384706 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.384707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:39.384851 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:39.385518 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:35:39.385526 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:35:39.387004 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:35:39.388795 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:35:39.388803 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:35:39.388805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:35:39.389041 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:35:39.389275 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:35:39.389275 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:35:39.389276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:35:39.389394 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:35:39.389661 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:35:39.389668 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:35:39.389813 (+ 145us) client_negotiation.cc:769] Sending connection context
0618 23:35:39.389930 (+ 117us) client_negotiation.cc:240] Negotiation successful
0618 23:35:39.390098 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":126,"threads_started":1}
I20260618 23:35:39.391430 19870 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40279
I20260618 23:35:39.391522 19870 heartbeater.cc:464] Registering TS with master...
I20260618 23:35:39.391749 19870 heartbeater.cc:511] Master 127.0.98.62:40279 requested a full tablet report, sending...
I20260618 23:35:39.392225 19545 ts_manager.cc:194] Registered new tserver with Master: ff62710620ff406a8eabc2a41e37ad33 (127.0.98.2:43531)
I20260618 23:35:39.392649 19545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36761
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260618 23:35:39.415875 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40279
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:39.510859 19875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:39.511212 19875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:39.511241 19875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:39.512588 19875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:39.512638 19875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:39.512667 19875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:35:39.514191 19875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:39.514408 19875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:39.514613 19875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:39.520504 19880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:39.520601 19875 server_base.cc:1034] running on GCE node
W20260618 23:35:39.520493 19881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:39.520529 19883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:39.521117 19875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:39.521440 19875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:39.522607 19875 hybrid_clock.cc:648] HybridClock initialized: now 1781825739522577 us; error 48 us; skew 500 ppm
Jun 18 23:35:39 dist-test-slave-6224 krb5kdc[19500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:39.525645 19875 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260618 23:35:39.526217 19875 webserver.cc:460] Webserver started at http://127.0.98.3:40433/ using document root <none> and password file <none>
I20260618 23:35:39.526430 19875 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:39.526475 19875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:39.526577 19875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:39.527843 19875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "a3b8abe99ac447c3a9888333742042ab"
format_stamp: "Formatted at 2026-06-18 23:35:39 on dist-test-slave-6224"
server_key: "47e201c43cc98bcf69b55c9fec3dc898"
server_key_iv: "4f567c4825f4d9d7a42bdfe11afbd1a4"
server_key_version: "encryptionkey@0"
I20260618 23:35:39.528185 19875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a3b8abe99ac447c3a9888333742042ab"
format_stamp: "Formatted at 2026-06-18 23:35:39 on dist-test-slave-6224"
server_key: "47e201c43cc98bcf69b55c9fec3dc898"
server_key_iv: "4f567c4825f4d9d7a42bdfe11afbd1a4"
server_key_version: "encryptionkey@0"
I20260618 23:35:39.529644 19875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:35:39.530592 19890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:39.530903 19875 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:39.531024 19875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "a3b8abe99ac447c3a9888333742042ab"
format_stamp: "Formatted at 2026-06-18 23:35:39 on dist-test-slave-6224"
server_key: "47e201c43cc98bcf69b55c9fec3dc898"
server_key_iv: "4f567c4825f4d9d7a42bdfe11afbd1a4"
server_key_version: "encryptionkey@0"
I20260618 23:35:39.531121 19875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:39.541265 19875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:39.543744 19875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:39.543866 19875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:39.544063 19875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:35:39.544422 19875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:35:39.544461 19875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:39.544521 19875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:35:39.544553 19875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:39.551194 19875 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43297
I20260618 23:35:39.551386 20003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43297 every 8 connection(s)
I20260618 23:35:39.551676 19875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
Jun 18 23:35:39 dist-test-slave-6224 krb5kdc[19500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:35:39.561967 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 19875
I20260618 23:35:39.562027 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.552627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36749 (local address 127.0.98.62:40279)
0618 23:35:39.552683 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:39.552685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:39.553030 (+ 345us) server_negotiation.cc:407] Connection header received
0618 23:35:39.554087 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:39.554088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:39.554102 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:39.554152 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:39.554819 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.555346 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.556462 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.556658 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.556746 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:39.559035 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:39.559037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:39.559037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:39.559044 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:39.560559 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:39.561009 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:39.561010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:39.561011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:39.561078 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:39.561334 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:39.561335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:39.561336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:39.561456 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:39.561540 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:39.561804 (+ 264us) server_negotiation.cc:299] Negotiation successful
0618 23:35:39.561931 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260618 23:35:39.562153 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:35:39.562170 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.552628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40279 (local address 127.0.98.3:36749)
0618 23:35:39.552883 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:39.552924 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:39.553951 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:39.554222 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:39.554226 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:39.554495 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:35:39.554705 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:39.554709 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.555446 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.555447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:39.556343 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:39.556346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.556779 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.556787 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:39.556901 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:39.557482 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:35:39.557490 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:35:39.558916 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:35:39.560685 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:35:39.560692 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:35:39.560695 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:35:39.560921 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:35:39.561155 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:35:39.561156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:35:39.561157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:35:39.561260 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:35:39.561531 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:35:39.561537 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:35:39.561669 (+ 132us) client_negotiation.cc:769] Sending connection context
0618 23:35:39.561784 (+ 115us) client_negotiation.cc:240] Negotiation successful
0618 23:35:39.561941 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":132,"threads_started":1}
I20260618 23:35:39.562388 392 external_mini_cluster.cc:1428] Setting key 6dc82bee16e3a1e5439f76b5c617e2b2
I20260618 23:35:39.562731 20004 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40279
I20260618 23:35:39.562888 20004 heartbeater.cc:464] Registering TS with master...
I20260618 23:35:39.563288 20004 heartbeater.cc:511] Master 127.0.98.62:40279 requested a full tablet report, sending...
I20260618 23:35:39.563848 19545 ts_manager.cc:194] Registered new tserver with Master: a3b8abe99ac447c3a9888333742042ab (127.0.98.3:43297)
I20260618 23:35:39.564291 19545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36749
I20260618 23:35:39.574865 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:35:39.583757 19584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.576664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48132 (local address 127.0.98.62:40279)
0618 23:35:39.576765 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:39.576768 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:39.576894 (+ 126us) server_negotiation.cc:407] Connection header received
0618 23:35:39.577018 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:39.577019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:39.577031 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:39.577113 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:39.577786 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.578359 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.579167 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.579380 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.579468 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:39.580924 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:39.580926 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:39.580927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:39.580935 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:39.582364 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:39.582761 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:39.582763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:39.582763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:39.582830 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:39.583055 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:39.583056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:39.583057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:39.583223 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:39.583341 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:39.583504 (+ 163us) server_negotiation.cc:299] Negotiation successful
0618 23:35:39.583637 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260618 23:35:39.585988 19545 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:35:39.588084 19545 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:35:39.596052 20016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.591377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46453 (local address 127.0.0.1:40618)
0618 23:35:39.592022 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:39.592032 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:39.592189 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:39.592566 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:39.592568 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:39.592571 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:39.592646 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:39.592647 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.593555 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.593556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:39.594871 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:39.594874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.595673 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.595674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:39.595774 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:39.595793 (+ 19us) client_negotiation.cc:769] Sending connection context
0618 23:35:39.595885 (+ 92us) client_negotiation.cc:240] Negotiation successful
0618 23:35:39.595961 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":624,"thread_start_us":141,"threads_started":1}
I20260618 23:35:39.596077 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.591542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.0.98.1:46453)
0618 23:35:39.591887 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:39.591893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:39.592161 (+ 268us) server_negotiation.cc:407] Connection header received
0618 23:35:39.592308 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:39.592311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:39.592329 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:39.592392 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:39.592702 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.593441 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.594984 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.595538 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.595675 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:39.595754 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:39.595921 (+ 167us) server_negotiation.cc:299] Negotiation successful
0618 23:35:39.595972 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":181,"threads_started":1}
I20260618 23:35:39.597121 19670 tablet_service.cc:1467] Processing CreateTablet for tablet f7719e1e6cc949a4bd3765b9aced451e (DEFAULT_TABLE table=test-table [id=b6cb1a7c65d2470a9055f7a5154b4372]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:35:39.597404 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.591393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43531 (local address 127.0.0.1:35348)
0618 23:35:39.591976 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:39.591987 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:39.592051 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:39.592421 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:39.592422 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:39.592423 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:39.592460 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:39.592462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.593927 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.593929 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:39.595498 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:39.595500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.597100 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.597101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:39.597184 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:39.597189 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:35:39.597263 (+ 74us) client_negotiation.cc:240] Negotiation successful
0618 23:35:39.597322 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":33,"thread_start_us":89,"threads_started":1}
I20260618 23:35:39.597476 20019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.591781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.0.98.2:43531)
0618 23:35:39.592155 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:39.592159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:39.592170 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:35:39.592209 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:39.592211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:39.592226 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:39.592330 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:39.592545 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.593790 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.595645 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.596992 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.597071 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:39.597130 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:39.597300 (+ 170us) server_negotiation.cc:299] Negotiation successful
0618 23:35:39.597379 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":110,"threads_started":1}
I20260618 23:35:39.597560 19670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7719e1e6cc949a4bd3765b9aced451e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:39.598400 19804 tablet_service.cc:1467] Processing CreateTablet for tablet f7719e1e6cc949a4bd3765b9aced451e (DEFAULT_TABLE table=test-table [id=b6cb1a7c65d2470a9055f7a5154b4372]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:35:39.598795 19804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7719e1e6cc949a4bd3765b9aced451e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:39.601266 20021 tablet_bootstrap.cc:492] T f7719e1e6cc949a4bd3765b9aced451e P 985ec866adfb43128d95efafe2293e69: Bootstrap starting.
I20260618 23:35:39.601903 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.591377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43297 (local address 127.0.0.1:42196)
0618 23:35:39.591753 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:39.591805 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:39.591895 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:39.592334 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:39.592338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:39.592341 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:39.592428 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:39.592430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.593929 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.593930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:39.595494 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:39.595498 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.601566 (+ 6068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.601567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:39.601652 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:39.601656 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:35:39.601746 (+ 90us) client_negotiation.cc:240] Negotiation successful
0618 23:35:39.601806 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":157,"thread_start_us":123,"threads_started":1}
I20260618 23:35:39.602181 20021 tablet_bootstrap.cc:654] T f7719e1e6cc949a4bd3765b9aced451e P 985ec866adfb43128d95efafe2293e69: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:39.601943 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.591574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.0.98.3:43297)
0618 23:35:39.592027 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:39.592032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:39.592047 (+ 15us) server_negotiation.cc:407] Connection header received
0618 23:35:39.592095 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:39.592098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:39.592119 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:39.592219 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:39.592524 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.593790 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.595634 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.601358 (+ 5724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.601451 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:39.601523 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:39.601800 (+ 277us) server_negotiation.cc:299] Negotiation successful
0618 23:35:39.601854 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":151,"threads_started":1}
I20260618 23:35:39.602667 20022 tablet_bootstrap.cc:492] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33: Bootstrap starting.
I20260618 23:35:39.602898 20021 log.cc:824] T f7719e1e6cc949a4bd3765b9aced451e P 985ec866adfb43128d95efafe2293e69: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:39.603236 19938 tablet_service.cc:1467] Processing CreateTablet for tablet f7719e1e6cc949a4bd3765b9aced451e (DEFAULT_TABLE table=test-table [id=b6cb1a7c65d2470a9055f7a5154b4372]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:35:39.603591 20022 tablet_bootstrap.cc:654] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:39.603684 19938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7719e1e6cc949a4bd3765b9aced451e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:39.603986 20021 tablet_bootstrap.cc:492] T f7719e1e6cc949a4bd3765b9aced451e P 985ec866adfb43128d95efafe2293e69: No bootstrap required, opened a new log
I20260618 23:35:39.604070 20021 ts_tablet_manager.cc:1397] T f7719e1e6cc949a4bd3765b9aced451e P 985ec866adfb43128d95efafe2293e69: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 23:35:39.604234 20022 log.cc:824] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:39.605078 20022 tablet_bootstrap.cc:492] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33: No bootstrap required, opened a new log
I20260618 23:35:39.605181 20022 ts_tablet_manager.cc:1397] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 23:35:39.606382 20021 raft_consensus.cc:348] T f7719e1e6cc949a4bd3765b9aced451e P 985ec866adfb43128d95efafe2293e69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985ec866adfb43128d95efafe2293e69" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46453 } } peers { permanent_uuid: "a3b8abe99ac447c3a9888333742042ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43297 } } peers { permanent_uuid: "ff62710620ff406a8eabc2a41e37ad33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43531 } }
I20260618 23:35:39.606590 20021 raft_consensus.cc:374] T f7719e1e6cc949a4bd3765b9aced451e P 985ec866adfb43128d95efafe2293e69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:39.606629 20021 raft_consensus.cc:729] T f7719e1e6cc949a4bd3765b9aced451e P 985ec866adfb43128d95efafe2293e69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 985ec866adfb43128d95efafe2293e69, State: Initialized, Role: FOLLOWER
I20260618 23:35:39.606798 20021 consensus_queue.cc:260] T f7719e1e6cc949a4bd3765b9aced451e P 985ec866adfb43128d95efafe2293e69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985ec866adfb43128d95efafe2293e69" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46453 } } peers { permanent_uuid: "a3b8abe99ac447c3a9888333742042ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43297 } } peers { permanent_uuid: "ff62710620ff406a8eabc2a41e37ad33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43531 } }
I20260618 23:35:39.607141 20021 ts_tablet_manager.cc:1428] T f7719e1e6cc949a4bd3765b9aced451e P 985ec866adfb43128d95efafe2293e69: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:35:39.606999 20022 raft_consensus.cc:348] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985ec866adfb43128d95efafe2293e69" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46453 } } peers { permanent_uuid: "a3b8abe99ac447c3a9888333742042ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43297 } } peers { permanent_uuid: "ff62710620ff406a8eabc2a41e37ad33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43531 } }
I20260618 23:35:39.607239 20022 raft_consensus.cc:374] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:39.607295 20022 raft_consensus.cc:729] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff62710620ff406a8eabc2a41e37ad33, State: Initialized, Role: FOLLOWER
I20260618 23:35:39.607340 19736 heartbeater.cc:503] Master 127.0.98.62:40279 was elected leader, sending a full tablet report...
I20260618 23:35:39.607402 20026 tablet_bootstrap.cc:492] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab: Bootstrap starting.
I20260618 23:35:39.607436 20022 consensus_queue.cc:260] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985ec866adfb43128d95efafe2293e69" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46453 } } peers { permanent_uuid: "a3b8abe99ac447c3a9888333742042ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43297 } } peers { permanent_uuid: "ff62710620ff406a8eabc2a41e37ad33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43531 } }
I20260618 23:35:39.607782 20022 ts_tablet_manager.cc:1428] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260618 23:35:39.607911 19870 heartbeater.cc:503] Master 127.0.98.62:40279 was elected leader, sending a full tablet report...
I20260618 23:35:39.608279 20026 tablet_bootstrap.cc:654] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:39.608938 20026 log.cc:824] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:39.609810 20026 tablet_bootstrap.cc:492] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab: No bootstrap required, opened a new log
I20260618 23:35:39.609872 20026 ts_tablet_manager.cc:1397] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 23:35:39.611259 20026 raft_consensus.cc:348] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985ec866adfb43128d95efafe2293e69" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46453 } } peers { permanent_uuid: "a3b8abe99ac447c3a9888333742042ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43297 } } peers { permanent_uuid: "ff62710620ff406a8eabc2a41e37ad33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43531 } }
I20260618 23:35:39.611402 20026 raft_consensus.cc:374] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:39.611424 20026 raft_consensus.cc:729] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3b8abe99ac447c3a9888333742042ab, State: Initialized, Role: FOLLOWER
I20260618 23:35:39.611558 20026 consensus_queue.cc:260] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985ec866adfb43128d95efafe2293e69" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46453 } } peers { permanent_uuid: "a3b8abe99ac447c3a9888333742042ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43297 } } peers { permanent_uuid: "ff62710620ff406a8eabc2a41e37ad33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43531 } }
I20260618 23:35:39.612013 20026 ts_tablet_manager.cc:1428] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:35:39.612187 20004 heartbeater.cc:503] Master 127.0.98.62:40279 was elected leader, sending a full tablet report...
W20260618 23:35:39.629678 19871 tablet.cc:2369] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:35:39.714800 19737 tablet.cc:2369] T f7719e1e6cc949a4bd3765b9aced451e P 985ec866adfb43128d95efafe2293e69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:35:39.769414 20027 raft_consensus.cc:482] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:35:39.769552 20027 raft_consensus.cc:504] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985ec866adfb43128d95efafe2293e69" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46453 } } peers { permanent_uuid: "a3b8abe99ac447c3a9888333742042ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43297 } } peers { permanent_uuid: "ff62710620ff406a8eabc2a41e37ad33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43531 } }
I20260618 23:35:39.769892 20027 leader_election.cc:288] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 985ec866adfb43128d95efafe2293e69 (127.0.98.1:46453), a3b8abe99ac447c3a9888333742042ab (127.0.98.3:43297)
I20260618 23:35:39.771572 20029 raft_consensus.cc:482] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:35:39.771692 20029 raft_consensus.cc:504] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985ec866adfb43128d95efafe2293e69" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46453 } } peers { permanent_uuid: "a3b8abe99ac447c3a9888333742042ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43297 } } peers { permanent_uuid: "ff62710620ff406a8eabc2a41e37ad33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43531 } }
I20260618 23:35:39.772058 20029 leader_election.cc:288] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 985ec866adfb43128d95efafe2293e69 (127.0.98.1:46453), ff62710620ff406a8eabc2a41e37ad33 (127.0.98.2:43531)
I20260618 23:35:39.773782 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.770200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46453 (local address 127.0.98.2:49961)
0618 23:35:39.770271 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:39.770290 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:39.770348 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:39.770556 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:39.770558 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:39.770560 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:39.770642 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:39.770644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.771459 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.771460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:39.772725 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:39.772727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.773516 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.773517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:39.773603 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:39.773607 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:35:39.773663 (+ 56us) client_negotiation.cc:240] Negotiation successful
0618 23:35:39.773707 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260618 23:35:39.774264 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.770277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49961 (local address 127.0.98.1:46453)
0618 23:35:39.770351 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:39.770353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:39.770367 (+ 14us) server_negotiation.cc:407] Connection header received
0618 23:35:39.770451 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:39.770452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:39.770462 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:39.770542 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:39.770875 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.771363 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.772836 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.773408 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.773474 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:39.773485 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:39.774163 (+ 678us) server_negotiation.cc:299] Negotiation successful
0618 23:35:39.774202 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260618 23:35:39.774343 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.770291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58695 (local address 127.0.98.3:43297)
0618 23:35:39.770343 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:39.770345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:39.770734 (+ 389us) server_negotiation.cc:407] Connection header received
0618 23:35:39.770763 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:39.770764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:39.770773 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:39.770835 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:39.771046 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.771791 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.773310 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.773921 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.773970 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:39.773981 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:39.774211 (+ 230us) server_negotiation.cc:299] Negotiation successful
0618 23:35:39.774255 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":23}
I20260618 23:35:39.774540 19690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7719e1e6cc949a4bd3765b9aced451e" candidate_uuid: "ff62710620ff406a8eabc2a41e37ad33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "985ec866adfb43128d95efafe2293e69" is_pre_election: true
I20260618 23:35:39.774657 20030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.770200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43297 (local address 127.0.98.2:58695)
0618 23:35:39.770609 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:39.770633 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:39.770691 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:39.770854 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:39.770855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:39.770857 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:39.770946 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:39.770947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.771907 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.771908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:39.773215 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:39.773216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.774061 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.774062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:39.774135 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:39.774139 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:35:39.774480 (+ 341us) client_negotiation.cc:240] Negotiation successful
0618 23:35:39.774530 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":200,"thread_start_us":152,"threads_started":1}
I20260618 23:35:39.774682 19690 raft_consensus.cc:2454] T f7719e1e6cc949a4bd3765b9aced451e P 985ec866adfb43128d95efafe2293e69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff62710620ff406a8eabc2a41e37ad33 in term 0.
I20260618 23:35:39.774973 19758 leader_election.cc:302] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 985ec866adfb43128d95efafe2293e69, ff62710620ff406a8eabc2a41e37ad33; no voters:
I20260618 23:35:39.775170 20027 raft_consensus.cc:2781] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:35:39.775255 20027 raft_consensus.cc:482] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:35:39.775283 20027 raft_consensus.cc:3037] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:39.775478 19958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7719e1e6cc949a4bd3765b9aced451e" candidate_uuid: "ff62710620ff406a8eabc2a41e37ad33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3b8abe99ac447c3a9888333742042ab" is_pre_election: true
I20260618 23:35:39.775687 19958 raft_consensus.cc:2454] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff62710620ff406a8eabc2a41e37ad33 in term 0.
I20260618 23:35:39.776103 20019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.772334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43915 (local address 127.0.98.2:43531)
0618 23:35:39.772435 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:39.772437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:39.772444 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:35:39.772463 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:39.772464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:39.772476 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:39.772520 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:39.772703 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.773348 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.774625 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.775126 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.775188 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:39.775197 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:39.775997 (+ 800us) server_negotiation.cc:299] Negotiation successful
0618 23:35:39.776046 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260618 23:35:39.776108 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.772242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43531 (local address 127.0.98.3:43915)
0618 23:35:39.772304 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:39.772314 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:39.772352 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:39.772557 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:39.772558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:39.772560 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:39.772633 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:39.772635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.773433 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.773433 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:39.774529 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:39.774532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.775844 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.775845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:39.775927 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:39.775931 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:35:39.775991 (+ 60us) client_negotiation.cc:240] Negotiation successful
0618 23:35:39.776040 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260618 23:35:39.776371 20031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.772501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46453 (local address 127.0.98.3:39053)
0618 23:35:39.772841 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:39.772853 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:39.772897 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:39.773097 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:39.773097 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:39.773099 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:39.773172 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:39.773174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.774052 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.774053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:39.775261 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:39.775264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.776062 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.776063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:39.776148 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:39.776154 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:35:39.776186 (+ 32us) client_negotiation.cc:240] Negotiation successful
0618 23:35:39.776234 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":87,"threads_started":1}
I20260618 23:35:39.776425 19824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7719e1e6cc949a4bd3765b9aced451e" candidate_uuid: "a3b8abe99ac447c3a9888333742042ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff62710620ff406a8eabc2a41e37ad33" is_pre_election: true
I20260618 23:35:39.776446 20027 raft_consensus.cc:504] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985ec866adfb43128d95efafe2293e69" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46453 } } peers { permanent_uuid: "a3b8abe99ac447c3a9888333742042ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43297 } } peers { permanent_uuid: "ff62710620ff406a8eabc2a41e37ad33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43531 } }
I20260618 23:35:39.776573 19824 raft_consensus.cc:2379] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a3b8abe99ac447c3a9888333742042ab in current term 1: Already voted for candidate ff62710620ff406a8eabc2a41e37ad33 in this term.
I20260618 23:35:39.776701 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.772671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39053 (local address 127.0.98.1:46453)
0618 23:35:39.772987 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:39.772989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:39.772995 (+ 6us) server_negotiation.cc:407] Connection header received
0618 23:35:39.773012 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:39.773013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:39.773020 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:39.773227 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:39.773252 (+ 25us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.773957 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.775404 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.775955 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.776548 (+ 593us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:39.776562 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:39.776583 (+ 21us) server_negotiation.cc:299] Negotiation successful
0618 23:35:39.776625 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":72,"threads_started":1}
I20260618 23:35:39.776721 19690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7719e1e6cc949a4bd3765b9aced451e" candidate_uuid: "ff62710620ff406a8eabc2a41e37ad33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "985ec866adfb43128d95efafe2293e69"
I20260618 23:35:39.776793 19690 raft_consensus.cc:3037] T f7719e1e6cc949a4bd3765b9aced451e P 985ec866adfb43128d95efafe2293e69 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:39.776984 19689 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7719e1e6cc949a4bd3765b9aced451e" candidate_uuid: "a3b8abe99ac447c3a9888333742042ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "985ec866adfb43128d95efafe2293e69" is_pre_election: true
I20260618 23:35:39.777343 19892 leader_election.cc:302] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3b8abe99ac447c3a9888333742042ab; no voters: 985ec866adfb43128d95efafe2293e69, ff62710620ff406a8eabc2a41e37ad33
I20260618 23:35:39.777496 20029 raft_consensus.cc:3037] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:39.777694 19690 raft_consensus.cc:2454] T f7719e1e6cc949a4bd3765b9aced451e P 985ec866adfb43128d95efafe2293e69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff62710620ff406a8eabc2a41e37ad33 in term 1.
I20260618 23:35:39.777906 19758 leader_election.cc:302] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 985ec866adfb43128d95efafe2293e69, ff62710620ff406a8eabc2a41e37ad33; no voters:
I20260618 23:35:39.777935 20027 leader_election.cc:288] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 [CANDIDATE]: Term 1 election: Requested vote from peers 985ec866adfb43128d95efafe2293e69 (127.0.98.1:46453), a3b8abe99ac447c3a9888333742042ab (127.0.98.3:43297)
I20260618 23:35:39.777977 20027 raft_consensus.cc:2781] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:39.778018 20027 raft_consensus.cc:686] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 [term 1 LEADER]: Becoming Leader. State: Replica: ff62710620ff406a8eabc2a41e37ad33, State: Running, Role: LEADER
I20260618 23:35:39.778108 20027 consensus_queue.cc:237] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985ec866adfb43128d95efafe2293e69" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46453 } } peers { permanent_uuid: "a3b8abe99ac447c3a9888333742042ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43297 } } peers { permanent_uuid: "ff62710620ff406a8eabc2a41e37ad33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43531 } }
I20260618 23:35:39.778676 20029 raft_consensus.cc:2726] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260618 23:35:39.778630 19958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7719e1e6cc949a4bd3765b9aced451e" candidate_uuid: "ff62710620ff406a8eabc2a41e37ad33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3b8abe99ac447c3a9888333742042ab"
I20260618 23:35:39.779163 19544 catalog_manager.cc:5495] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff62710620ff406a8eabc2a41e37ad33 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ff62710620ff406a8eabc2a41e37ad33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "985ec866adfb43128d95efafe2293e69" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46453 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3b8abe99ac447c3a9888333742042ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff62710620ff406a8eabc2a41e37ad33" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43531 } health_report { overall_health: HEALTHY } } }
I20260618 23:35:39.779529 19958 raft_consensus.cc:2454] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff62710620ff406a8eabc2a41e37ad33 in term 1.
W20260618 23:35:39.802521 20005 tablet.cc:2369] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:35:39.830741 20019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.827704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.0.98.2:43531)
0618 23:35:39.827811 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:39.827813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:39.827826 (+ 13us) server_negotiation.cc:407] Connection header received
0618 23:35:39.827898 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:39.827899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:39.827925 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:39.828030 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:35:39.828352 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.828908 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.829809 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:39.830021 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:39.830115 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:39.830312 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:35:39.830452 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:35:39.830578 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:39.830596 (+ 18us) server_negotiation.cc:299] Negotiation successful
0618 23:35:39.830621 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260618 23:35:39.833431 19958 raft_consensus.cc:1261] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab [term 1 FOLLOWER]: Refusing update from remote peer ff62710620ff406a8eabc2a41e37ad33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:35:39.833907 19690 raft_consensus.cc:1261] T f7719e1e6cc949a4bd3765b9aced451e P 985ec866adfb43128d95efafe2293e69 [term 1 FOLLOWER]: Refusing update from remote peer ff62710620ff406a8eabc2a41e37ad33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:35:39.833912 20027 consensus_queue.cc:1035] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3b8abe99ac447c3a9888333742042ab" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:35:39.834334 20027 consensus_queue.cc:1035] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "985ec866adfb43128d95efafe2293e69" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:35:39.836867 20036 mvcc.cc:205] Tried to move back new op lower bound from 7298358230353731584 to 7298358230131732480. Current Snapshot: MvccSnapshot[applied={T|T < 7298358230353731584}]
I20260618 23:35:39.837052 20038 mvcc.cc:205] Tried to move back new op lower bound from 7298358230353731584 to 7298358230131732480. Current Snapshot: MvccSnapshot[applied={T|T < 7298358230353731584}]
I20260618 23:35:39.838002 20037 mvcc.cc:205] Tried to move back new op lower bound from 7298358230353731584 to 7298358230131732480. Current Snapshot: MvccSnapshot[applied={T|T < 7298358230353731584 or (T in {7298358230353731584})}]
I20260618 23:35:39.840457 19545 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:35:39.840583 19545 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:35:39.841631 19545 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3eed096eca414dbd9f7b553ba4c1fa43: Sending DeleteTablet for 3 replicas of tablet f7719e1e6cc949a4bd3765b9aced451e
I20260618 23:35:39.842070 19670 tablet_service.cc:1514] Processing DeleteTablet for tablet f7719e1e6cc949a4bd3765b9aced451e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:35:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40618
I20260618 23:35:39.842092 19938 tablet_service.cc:1514] Processing DeleteTablet for tablet f7719e1e6cc949a4bd3765b9aced451e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:35:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42196
I20260618 23:35:39.842270 19804 tablet_service.cc:1514] Processing DeleteTablet for tablet f7719e1e6cc949a4bd3765b9aced451e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:35:39 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35348
I20260618 23:35:39.842389 20047 tablet_replica.cc:330] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab: stopping tablet replica
I20260618 23:35:39.842497 20047 raft_consensus.cc:2229] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:35:39.842583 20048 tablet_replica.cc:330] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33: stopping tablet replica
I20260618 23:35:39.842603 20047 raft_consensus.cc:2258] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:35:39.842690 20048 raft_consensus.cc:2229] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:35:39.842837 20048 raft_consensus.cc:2258] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:35:39.843127 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 19607
I20260618 23:35:39.843225 20047 ts_tablet_manager.cc:1905] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:35:39.843381 20048 ts_tablet_manager.cc:1905] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:35:39.845093 20047 ts_tablet_manager.cc:1918] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:35:39.845160 20047 log.cc:1196] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/f7719e1e6cc949a4bd3765b9aced451e
I20260618 23:35:39.845489 20047 ts_tablet_manager.cc:1939] T f7719e1e6cc949a4bd3765b9aced451e P a3b8abe99ac447c3a9888333742042ab: Deleting consensus metadata
I20260618 23:35:39.845716 20048 ts_tablet_manager.cc:1918] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:35:39.845788 20048 log.cc:1196] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825489608381-392-0/minicluster-data/ts-1/wal/wals/f7719e1e6cc949a4bd3765b9aced451e
I20260618 23:35:39.845940 19532 catalog_manager.cc:4841] TS a3b8abe99ac447c3a9888333742042ab (127.0.98.3:43297): tablet f7719e1e6cc949a4bd3765b9aced451e (table test-table [id=b6cb1a7c65d2470a9055f7a5154b4372]) successfully deleted
I20260618 23:35:39.846083 20048 ts_tablet_manager.cc:1939] T f7719e1e6cc949a4bd3765b9aced451e P ff62710620ff406a8eabc2a41e37ad33: Deleting consensus metadata
I20260618 23:35:39.847781 19531 catalog_manager.cc:4841] TS ff62710620ff406a8eabc2a41e37ad33 (127.0.98.2:43531): tablet f7719e1e6cc949a4bd3765b9aced451e (table test-table [id=b6cb1a7c65d2470a9055f7a5154b4372]) successfully deleted
W20260618 23:35:39.852402 19530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46453 (error 108)
I20260618 23:35:39.852507 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 19741
I20260618 23:35:39.854766 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:39.854567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46453 (local address 127.0.0.1:40632)
0618 23:35:39.854671 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:39.854724 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46453: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260618 23:35:39.854945 19530 catalog_manager.cc:4584] TS 985ec866adfb43128d95efafe2293e69 (127.0.98.1:46453): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f7719e1e6cc949a4bd3765b9aced451e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46453: connect: Connection refused (error 111)
I20260618 23:35:39.857708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 19875
I20260618 23:35:39.864178 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 19513
2026-06-18T23:35:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2230 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-18T23:35:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:35:39Z Disabled control of system clock
I20260618 23:35:39.894829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42403
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 23:35:39.980165 20053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:39.980401 20053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:39.980423 20053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:39.981606 20053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:35:39.981655 20053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:39.981668 20053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:39.981680 20053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:35:39.981693 20053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:35:39.983069 20053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42403
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:39.983341 20053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:39.983697 20053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:39.989250 20061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:39.989262 20058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:39.989646 20053 server_base.cc:1034] running on GCE node
W20260618 23:35:39.989321 20059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:39.990026 20053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:39.990381 20053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:39.991598 20053 hybrid_clock.cc:648] HybridClock initialized: now 1781825739991579 us; error 50 us; skew 500 ppm
I20260618 23:35:39.992964 20053 webserver.cc:460] Webserver started at http://127.0.98.62:40477/ using document root <none> and password file <none>
I20260618 23:35:39.993251 20053 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:39.993341 20053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:39.993484 20053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:39.994553 20053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "bcf928a769db4e998fb33d66b4ac9d2c"
format_stamp: "Formatted at 2026-06-18 23:35:39 on dist-test-slave-6224"
server_key: "5c8e23ca6635ba4f0289a8379a9e6741"
server_key_iv: "7ed9a014200bbefbbec9e61fb978b10a"
server_key_version: "encryptionkey@0"
I20260618 23:35:39.994932 20053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "bcf928a769db4e998fb33d66b4ac9d2c"
format_stamp: "Formatted at 2026-06-18 23:35:39 on dist-test-slave-6224"
server_key: "5c8e23ca6635ba4f0289a8379a9e6741"
server_key_iv: "7ed9a014200bbefbbec9e61fb978b10a"
server_key_version: "encryptionkey@0"
I20260618 23:35:39.997320 20053 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260618 23:35:39.998225 20067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:39.998484 20053 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:39.998591 20053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "bcf928a769db4e998fb33d66b4ac9d2c"
format_stamp: "Formatted at 2026-06-18 23:35:39 on dist-test-slave-6224"
server_key: "5c8e23ca6635ba4f0289a8379a9e6741"
server_key_iv: "7ed9a014200bbefbbec9e61fb978b10a"
server_key_version: "encryptionkey@0"
I20260618 23:35:39.998682 20053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:40.011453 20053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:40.011857 20053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:40.012027 20053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:40.016841 20053 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42403
I20260618 23:35:40.016928 20119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42403 every 8 connection(s)
I20260618 23:35:40.017275 20053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:35:40.018481 20120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:40.020526 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 20053
I20260618 23:35:40.020659 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:35:40.020917 392 external_mini_cluster.cc:1428] Setting key 76a409e04c1f906528a3821db0b44d6b
I20260618 23:35:40.021871 20120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c: Bootstrap starting.
I20260618 23:35:40.022560 20120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:40.022930 20120 log.cc:824] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:40.024026 20120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c: No bootstrap required, opened a new log
I20260618 23:35:40.027029 20120 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf928a769db4e998fb33d66b4ac9d2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42403 } }
I20260618 23:35:40.027259 20120 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:40.027302 20120 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf928a769db4e998fb33d66b4ac9d2c, State: Initialized, Role: FOLLOWER
I20260618 23:35:40.027443 20120 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf928a769db4e998fb33d66b4ac9d2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42403 } }
I20260618 23:35:40.027529 20120 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:35:40.027588 20120 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:35:40.027649 20120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:40.028644 20120 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf928a769db4e998fb33d66b4ac9d2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42403 } }
I20260618 23:35:40.028762 20120 leader_election.cc:302] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcf928a769db4e998fb33d66b4ac9d2c; no voters:
I20260618 23:35:40.028940 20123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:40.021626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.0.98.62:42403)
0618 23:35:40.021850 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:40.021861 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:40.021892 (+ 31us) server_negotiation.cc:407] Connection header received
0618 23:35:40.022566 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:40.022573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:40.022687 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:40.022840 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:40.023213 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.024427 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.025644 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.026231 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.026343 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:40.027901 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:40.027907 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:40.027907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:40.028156 (+ 249us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:40.028209 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:40.028226 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:40.028293 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:40.028536 (+ 243us) server_negotiation.cc:299] Negotiation successful
0618 23:35:40.028658 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260618 23:35:40.028995 20120 leader_election.cc:288] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:35:40.029287 20126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:40.029356 20120 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:35:40.029896 20126 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c [term 1 LEADER]: Becoming Leader. State: Replica: bcf928a769db4e998fb33d66b4ac9d2c, State: Running, Role: LEADER
I20260618 23:35:40.030012 20126 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf928a769db4e998fb33d66b4ac9d2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42403 } }
I20260618 23:35:40.030495 20127 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcf928a769db4e998fb33d66b4ac9d2c. Latest consensus state: current_term: 1 leader_uuid: "bcf928a769db4e998fb33d66b4ac9d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf928a769db4e998fb33d66b4ac9d2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42403 } } }
I20260618 23:35:40.030593 20127 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:40.031278 20125 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcf928a769db4e998fb33d66b4ac9d2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcf928a769db4e998fb33d66b4ac9d2c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42403 } } }
I20260618 23:35:40.031577 20125 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c [sys.catalog]: This master's current role is: LEADER
W20260618 23:35:40.032254 20141 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:35:40.032326 20141 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:35:40.032409 20134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:35:40.032645 20134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:35:40.034361 20134 catalog_manager.cc:1347] Generated new cluster ID: a025187e242d4e40be35ad4837fbcfc4
I20260618 23:35:40.034412 20134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:35:40.045538 20134 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:35:40.046144 20134 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:35:40.049885 20134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c: Generated new TSK 0
I20260618 23:35:40.050173 20134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:35:40.058483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42403
--builtin_ntp_servers=127.0.98.20:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 23:35:40.153294 20144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:40.153566 20144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:40.153601 20144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:40.154755 20144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:40.154834 20144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:40.154878 20144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260618 23:35:40.156381 20144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:40.156659 20144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:40.156966 20144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:40.162483 20150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:40.162547 20144 server_base.cc:1034] running on GCE node
W20260618 23:35:40.162627 20152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:40.162485 20149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:40.163013 20144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:40.163316 20144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:40.164476 20144 hybrid_clock.cc:648] HybridClock initialized: now 1781825740164448 us; error 47 us; skew 500 ppm
I20260618 23:35:40.165922 20144 webserver.cc:460] Webserver started at http://127.0.98.1:33835/ using document root <none> and password file <none>
I20260618 23:35:40.166213 20144 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:40.166296 20144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:40.166469 20144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:40.167660 20144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-0/data/instance:
uuid: "f6fb60aba59c4524bb710074f0b90809"
format_stamp: "Formatted at 2026-06-18 23:35:40 on dist-test-slave-6224"
server_key: "196a3abd1d500fccde878754508a5c05"
server_key_iv: "804ddd569588fa2ac38d30f80dd8c33f"
server_key_version: "encryptionkey@0"
I20260618 23:35:40.168052 20144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f6fb60aba59c4524bb710074f0b90809"
format_stamp: "Formatted at 2026-06-18 23:35:40 on dist-test-slave-6224"
server_key: "196a3abd1d500fccde878754508a5c05"
server_key_iv: "804ddd569588fa2ac38d30f80dd8c33f"
server_key_version: "encryptionkey@0"
I20260618 23:35:40.169421 20144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260618 23:35:40.170222 20158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:40.170559 20144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:35:40.170727 20144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-0/wal
uuid: "f6fb60aba59c4524bb710074f0b90809"
format_stamp: "Formatted at 2026-06-18 23:35:40 on dist-test-slave-6224"
server_key: "196a3abd1d500fccde878754508a5c05"
server_key_iv: "804ddd569588fa2ac38d30f80dd8c33f"
server_key_version: "encryptionkey@0"
I20260618 23:35:40.170832 20144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:40.190086 20144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:40.190529 20144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:40.190703 20144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:40.191016 20144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:35:40.191458 20144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:35:40.191509 20144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:40.191596 20144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:35:40.191622 20144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:40.201690 20144 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40739
I20260618 23:35:40.201807 20271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40739 every 8 connection(s)
I20260618 23:35:40.202281 20144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-0/data/info.pb
I20260618 23:35:40.206458 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 20144
I20260618 23:35:40.206614 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-0/wal/instance
I20260618 23:35:40.206861 392 external_mini_cluster.cc:1428] Setting key 33401097377a25e6f4adad7e7aa0762f
I20260618 23:35:40.208089 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42403
--builtin_ntp_servers=127.0.98.20:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 23:35:40.209551 20123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:40.203288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57595 (local address 127.0.98.62:42403)
0618 23:35:40.203380 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:40.203384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:40.203849 (+ 465us) server_negotiation.cc:407] Connection header received
0618 23:35:40.204450 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:40.204451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:40.204469 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:40.204550 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:40.205526 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.206077 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.207894 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.208085 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.208149 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:40.209241 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:40.209242 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:40.209242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:40.209246 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:40.209261 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:40.209265 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:40.209315 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:40.209452 (+ 137us) server_negotiation.cc:299] Negotiation successful
0618 23:35:40.209479 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 23:35:40.209909 20274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:40.203229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42403 (local address 127.0.98.1:57595)
0618 23:35:40.203667 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:40.203721 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:40.204343 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:40.204644 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:40.204648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:40.204657 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:35:40.205404 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:40.205411 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.206379 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.206381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:40.207797 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:40.207801 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.208208 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.208209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:40.208403 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:40.209038 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:35:40.209039 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:35:40.209096 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:35:40.209133 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:35:40.209348 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:35:40.209355 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:35:40.209359 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:35:40.209444 (+ 85us) client_negotiation.cc:240] Negotiation successful
0618 23:35:40.209584 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":161,"threads_started":1}
I20260618 23:35:40.210541 20272 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42403
I20260618 23:35:40.210655 20272 heartbeater.cc:464] Registering TS with master...
I20260618 23:35:40.210917 20272 heartbeater.cc:511] Master 127.0.98.62:42403 requested a full tablet report, sending...
I20260618 23:35:40.211699 20084 ts_manager.cc:194] Registered new tserver with Master: f6fb60aba59c4524bb710074f0b90809 (127.0.98.1:40739)
I20260618 23:35:40.212675 20084 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:57595
W20260618 23:35:40.291504 20275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:40.291838 20275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:40.291885 20275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:40.293090 20275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:40.293171 20275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:40.293218 20275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260618 23:35:40.294667 20275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:40.294953 20275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:40.295270 20275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:40.301139 20281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:40.301146 20280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:40.301137 20283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:40.301160 20275 server_base.cc:1034] running on GCE node
I20260618 23:35:40.301668 20275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:40.302001 20275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:40.303176 20275 hybrid_clock.cc:648] HybridClock initialized: now 1781825740303170 us; error 32 us; skew 500 ppm
I20260618 23:35:40.304617 20275 webserver.cc:460] Webserver started at http://127.0.98.2:46583/ using document root <none> and password file <none>
I20260618 23:35:40.304877 20275 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:40.304929 20275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:40.305091 20275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:40.306192 20275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-1/data/instance:
uuid: "78fce8a82f304e2caaf12a69f07ba614"
format_stamp: "Formatted at 2026-06-18 23:35:40 on dist-test-slave-6224"
server_key: "3e4c928070d180867ed79eeff13a2f41"
server_key_iv: "bec27ead874bd5bfd0a452dd8817916c"
server_key_version: "encryptionkey@0"
I20260618 23:35:40.306564 20275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-1/wal/instance:
uuid: "78fce8a82f304e2caaf12a69f07ba614"
format_stamp: "Formatted at 2026-06-18 23:35:40 on dist-test-slave-6224"
server_key: "3e4c928070d180867ed79eeff13a2f41"
server_key_iv: "bec27ead874bd5bfd0a452dd8817916c"
server_key_version: "encryptionkey@0"
I20260618 23:35:40.308082 20275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:35:40.308943 20289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:40.309207 20275 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:40.309331 20275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-1/wal
uuid: "78fce8a82f304e2caaf12a69f07ba614"
format_stamp: "Formatted at 2026-06-18 23:35:40 on dist-test-slave-6224"
server_key: "3e4c928070d180867ed79eeff13a2f41"
server_key_iv: "bec27ead874bd5bfd0a452dd8817916c"
server_key_version: "encryptionkey@0"
I20260618 23:35:40.309432 20275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:40.316094 20275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:40.316376 20275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:40.316532 20275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:40.316756 20275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:35:40.317081 20275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:35:40.317145 20275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:40.317180 20275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:35:40.317209 20275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:40.324159 20275 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46423
I20260618 23:35:40.324245 20402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46423 every 8 connection(s)
I20260618 23:35:40.324633 20275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-1/data/info.pb
I20260618 23:35:40.330942 20123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:40.325684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46823 (local address 127.0.98.62:42403)
0618 23:35:40.325765 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:40.325767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:40.326135 (+ 368us) server_negotiation.cc:407] Connection header received
0618 23:35:40.326644 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:40.326645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:40.326659 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:40.326731 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:40.327408 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.328062 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.329077 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.329387 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.329448 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:40.330568 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:40.330570 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:40.330570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:40.330577 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:40.330601 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:40.330605 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:40.330672 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:40.330773 (+ 101us) server_negotiation.cc:299] Negotiation successful
0618 23:35:40.330799 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 23:35:40.331157 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:40.325665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42403 (local address 127.0.98.2:46823)
0618 23:35:40.325967 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:40.326004 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:40.326512 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:40.326787 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:40.326791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:40.326799 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:35:40.327296 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:40.327299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.328178 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.328179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:40.328983 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:40.328985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.329472 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.329473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:40.329586 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:40.330286 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:35:40.330287 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:35:40.330314 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:35:40.330337 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:35:40.330685 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:35:40.330689 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:35:40.330692 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:35:40.330752 (+ 60us) client_negotiation.cc:240] Negotiation successful
0618 23:35:40.330894 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":159,"threads_started":1}
I20260618 23:35:40.331822 20403 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42403
I20260618 23:35:40.331962 20403 heartbeater.cc:464] Registering TS with master...
I20260618 23:35:40.332139 20403 heartbeater.cc:511] Master 127.0.98.62:42403 requested a full tablet report, sending...
I20260618 23:35:40.332545 20084 ts_manager.cc:194] Registered new tserver with Master: 78fce8a82f304e2caaf12a69f07ba614 (127.0.98.2:46423)
I20260618 23:35:40.332926 20084 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:46823
I20260618 23:35:40.334672 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 20275
I20260618 23:35:40.334793 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-1/wal/instance
I20260618 23:35:40.335106 392 external_mini_cluster.cc:1428] Setting key 1466b8aa5afbaaac54fdb4c5db10056b
I20260618 23:35:40.336323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42403
--builtin_ntp_servers=127.0.98.20:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 23:35:40.427479 20406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:40.427772 20406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:40.427798 20406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:40.428968 20406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:40.429013 20406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:40.429039 20406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260618 23:35:40.430580 20406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:40.430909 20406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:40.431319 20406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:40.437529 20412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:40.437529 20411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:40.437702 20406 server_base.cc:1034] running on GCE node
W20260618 23:35:40.437582 20414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:40.438112 20406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:40.438459 20406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:40.439633 20406 hybrid_clock.cc:648] HybridClock initialized: now 1781825740439585 us; error 79 us; skew 500 ppm
I20260618 23:35:40.441093 20406 webserver.cc:460] Webserver started at http://127.0.98.3:40339/ using document root <none> and password file <none>
I20260618 23:35:40.441404 20406 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:40.441484 20406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:40.441656 20406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:40.442884 20406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-2/data/instance:
uuid: "412f6e1f833a4715b211460aba9dc69a"
format_stamp: "Formatted at 2026-06-18 23:35:40 on dist-test-slave-6224"
server_key: "27e47766020fe684fe711d3ab48573a0"
server_key_iv: "74a54cdeb8003425604c8f218d4692dc"
server_key_version: "encryptionkey@0"
I20260618 23:35:40.443274 20406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-2/wal/instance:
uuid: "412f6e1f833a4715b211460aba9dc69a"
format_stamp: "Formatted at 2026-06-18 23:35:40 on dist-test-slave-6224"
server_key: "27e47766020fe684fe711d3ab48573a0"
server_key_iv: "74a54cdeb8003425604c8f218d4692dc"
server_key_version: "encryptionkey@0"
I20260618 23:35:40.444839 20406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:35:40.445771 20420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:40.446095 20406 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:40.446233 20406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-2/wal
uuid: "412f6e1f833a4715b211460aba9dc69a"
format_stamp: "Formatted at 2026-06-18 23:35:40 on dist-test-slave-6224"
server_key: "27e47766020fe684fe711d3ab48573a0"
server_key_iv: "74a54cdeb8003425604c8f218d4692dc"
server_key_version: "encryptionkey@0"
I20260618 23:35:40.446341 20406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:40.468549 20406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:40.468914 20406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:40.469089 20406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:40.469354 20406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:35:40.469744 20406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:35:40.469810 20406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:40.469849 20406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:35:40.469880 20406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:40.476966 20406 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38191
I20260618 23:35:40.477011 20533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38191 every 8 connection(s)
I20260618 23:35:40.477362 20406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-2/data/info.pb
I20260618 23:35:40.483011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 20406
I20260618 23:35:40.483186 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-2/wal/instance
I20260618 23:35:40.483455 392 external_mini_cluster.cc:1428] Setting key 0dce5d4c2825ccaed45b37109eaf598a
I20260618 23:35:40.483491 20123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:40.478344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41647 (local address 127.0.98.62:42403)
0618 23:35:40.478452 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:40.478455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:40.478784 (+ 329us) server_negotiation.cc:407] Connection header received
0618 23:35:40.479392 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:40.479393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:40.479408 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:40.479475 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:40.480251 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.480769 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.481768 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.481962 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.482044 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:40.483074 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:40.483075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:40.483076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:40.483080 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:40.483100 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:40.483109 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:40.483178 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:40.483337 (+ 159us) server_negotiation.cc:299] Negotiation successful
0618 23:35:40.483390 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260618 23:35:40.483799 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:40.478336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42403 (local address 127.0.98.3:41647)
0618 23:35:40.478634 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:40.478675 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:40.479255 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:40.479625 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:40.479628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:40.479637 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:35:40.480156 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:40.480159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.480882 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.480883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:40.481660 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:40.481662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.482063 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.482064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:40.482194 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:40.482911 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:35:40.482911 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:35:40.482939 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:35:40.482962 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:35:40.483179 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:35:40.483185 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:35:40.483188 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:35:40.483284 (+ 96us) client_negotiation.cc:240] Negotiation successful
0618 23:35:40.483396 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":151,"threads_started":1}
I20260618 23:35:40.484582 20534 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42403
I20260618 23:35:40.484736 20534 heartbeater.cc:464] Registering TS with master...
I20260618 23:35:40.484990 20534 heartbeater.cc:511] Master 127.0.98.62:42403 requested a full tablet report, sending...
I20260618 23:35:40.485466 20084 ts_manager.cc:194] Registered new tserver with Master: 412f6e1f833a4715b211460aba9dc69a (127.0.98.3:38191)
I20260618 23:35:40.486073 20084 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:41647
I20260618 23:35:40.496253 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:35:40.501533 20123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:40.497899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48412 (local address 127.0.98.62:42403)
0618 23:35:40.497985 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:40.497987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:40.498216 (+ 229us) server_negotiation.cc:407] Connection header received
0618 23:35:40.498293 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:40.498294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:40.498307 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:40.498366 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:40.498544 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.499081 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.499999 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.500246 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.500334 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:40.501212 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:40.501214 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:40.501214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:40.501218 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:40.501234 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:40.501238 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:40.501299 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:40.501368 (+ 69us) server_negotiation.cc:299] Negotiation successful
0618 23:35:40.501392 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 23:35:40.503310 20084 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:35:40.505317 20084 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:35:40.513266 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:40.508247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40739 (local address 127.0.0.1:53810)
0618 23:35:40.508683 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:40.508697 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:40.508770 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:40.509353 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:40.509354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:40.509356 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:40.509405 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:40.509407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.510968 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.510970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:40.512028 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:40.512032 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.512946 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.512947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:40.513034 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:40.513040 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:35:40.513075 (+ 35us) client_negotiation.cc:240] Negotiation successful
0618 23:35:40.513157 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"mutex_wait_us":3,"thread_start_us":84,"threads_started":1}
I20260618 23:35:40.513311 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:40.508618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50078 (local address 127.0.98.3:38191)
0618 23:35:40.508897 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:40.508902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:40.509232 (+ 330us) server_negotiation.cc:407] Connection header received
0618 23:35:40.509321 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:40.509323 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:40.509338 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:40.509421 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:40.509528 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.510703 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.512104 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.512777 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.512868 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:40.512987 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:40.513118 (+ 131us) server_negotiation.cc:299] Negotiation successful
0618 23:35:40.513173 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":104,"threads_started":1}
I20260618 23:35:40.513603 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:40.508248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46423 (local address 127.0.0.1:46654)
0618 23:35:40.508462 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:40.508510 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:40.508592 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:40.509192 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:40.509195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:40.509198 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:40.509292 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:40.509295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.510426 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.510427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:40.511841 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:40.511845 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.513364 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.513365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:40.513436 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:40.513440 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:35:40.513482 (+ 42us) client_negotiation.cc:240] Negotiation successful
0618 23:35:40.513528 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
I20260618 23:35:40.513701 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:40.508941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53810 (local address 127.0.98.1:40739)
0618 23:35:40.509165 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:40.509170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:40.509180 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:35:40.509219 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:40.509220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:40.509236 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:40.509347 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:40.509481 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.510875 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.512133 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.512854 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.513456 (+ 602us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:40.513548 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:40.513574 (+ 26us) server_negotiation.cc:299] Negotiation successful
0618 23:35:40.513609 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":98,"threads_started":1}
I20260618 23:35:40.513823 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:40.508497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38191 (local address 127.0.0.1:50078)
0618 23:35:40.509155 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:40.509166 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:40.509234 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:40.509419 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:40.509419 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:40.509421 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:40.509467 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:40.509468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.510816 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.510817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:40.511997 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:40.512005 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.512880 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.512881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:40.512962 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:40.512976 (+ 14us) client_negotiation.cc:769] Sending connection context
0618 23:35:40.513528 (+ 552us) client_negotiation.cc:240] Negotiation successful
0618 23:35:40.513601 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"mutex_wait_us":47,"thread_start_us":118,"threads_started":1}
I20260618 23:35:40.513895 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:40.508702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46654 (local address 127.0.98.2:46423)
0618 23:35:40.508953 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:40.508958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:40.508969 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:35:40.509015 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:40.509017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:40.509035 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:40.509136 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:40.509504 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.510320 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.511961 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.512829 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.512926 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:40.513740 (+ 814us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:40.513775 (+ 35us) server_negotiation.cc:299] Negotiation successful
0618 23:35:40.513821 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":109,"threads_started":1}
I20260618 23:35:40.514385 20206 tablet_service.cc:1467] Processing CreateTablet for tablet c3b8a989deb04921b91760396d7f3912 (DEFAULT_TABLE table=test-table [id=101f702c1f14468687410285c4badea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:35:40.514549 20468 tablet_service.cc:1467] Processing CreateTablet for tablet c3b8a989deb04921b91760396d7f3912 (DEFAULT_TABLE table=test-table [id=101f702c1f14468687410285c4badea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:35:40.514782 20206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b8a989deb04921b91760396d7f3912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:40.514838 20468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b8a989deb04921b91760396d7f3912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:40.514926 20337 tablet_service.cc:1467] Processing CreateTablet for tablet c3b8a989deb04921b91760396d7f3912 (DEFAULT_TABLE table=test-table [id=101f702c1f14468687410285c4badea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:35:40.515372 20337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3b8a989deb04921b91760396d7f3912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:40.518379 20548 tablet_bootstrap.cc:492] T c3b8a989deb04921b91760396d7f3912 P 412f6e1f833a4715b211460aba9dc69a: Bootstrap starting.
I20260618 23:35:40.519166 20548 tablet_bootstrap.cc:654] T c3b8a989deb04921b91760396d7f3912 P 412f6e1f833a4715b211460aba9dc69a: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:40.519436 20548 log.cc:824] T c3b8a989deb04921b91760396d7f3912 P 412f6e1f833a4715b211460aba9dc69a: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:40.519757 20550 tablet_bootstrap.cc:492] T c3b8a989deb04921b91760396d7f3912 P 78fce8a82f304e2caaf12a69f07ba614: Bootstrap starting.
I20260618 23:35:40.519788 20549 tablet_bootstrap.cc:492] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809: Bootstrap starting.
I20260618 23:35:40.520318 20548 tablet_bootstrap.cc:492] T c3b8a989deb04921b91760396d7f3912 P 412f6e1f833a4715b211460aba9dc69a: No bootstrap required, opened a new log
I20260618 23:35:40.520404 20548 ts_tablet_manager.cc:1397] T c3b8a989deb04921b91760396d7f3912 P 412f6e1f833a4715b211460aba9dc69a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260618 23:35:40.520452 20549 tablet_bootstrap.cc:654] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:40.520681 20550 tablet_bootstrap.cc:654] T c3b8a989deb04921b91760396d7f3912 P 78fce8a82f304e2caaf12a69f07ba614: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:40.520787 20549 log.cc:824] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:40.521114 20550 log.cc:824] T c3b8a989deb04921b91760396d7f3912 P 78fce8a82f304e2caaf12a69f07ba614: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:40.521627 20549 tablet_bootstrap.cc:492] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809: No bootstrap required, opened a new log
I20260618 23:35:40.521708 20549 ts_tablet_manager.cc:1397] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:35:40.522078 20550 tablet_bootstrap.cc:492] T c3b8a989deb04921b91760396d7f3912 P 78fce8a82f304e2caaf12a69f07ba614: No bootstrap required, opened a new log
I20260618 23:35:40.522164 20550 ts_tablet_manager.cc:1397] T c3b8a989deb04921b91760396d7f3912 P 78fce8a82f304e2caaf12a69f07ba614: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:35:40.523516 20548 raft_consensus.cc:348] T c3b8a989deb04921b91760396d7f3912 P 412f6e1f833a4715b211460aba9dc69a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78fce8a82f304e2caaf12a69f07ba614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46423 } } peers { permanent_uuid: "f6fb60aba59c4524bb710074f0b90809" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40739 } } peers { permanent_uuid: "412f6e1f833a4715b211460aba9dc69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38191 } }
I20260618 23:35:40.523736 20548 raft_consensus.cc:374] T c3b8a989deb04921b91760396d7f3912 P 412f6e1f833a4715b211460aba9dc69a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:40.523787 20548 raft_consensus.cc:729] T c3b8a989deb04921b91760396d7f3912 P 412f6e1f833a4715b211460aba9dc69a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 412f6e1f833a4715b211460aba9dc69a, State: Initialized, Role: FOLLOWER
I20260618 23:35:40.523896 20548 consensus_queue.cc:260] T c3b8a989deb04921b91760396d7f3912 P 412f6e1f833a4715b211460aba9dc69a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78fce8a82f304e2caaf12a69f07ba614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46423 } } peers { permanent_uuid: "f6fb60aba59c4524bb710074f0b90809" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40739 } } peers { permanent_uuid: "412f6e1f833a4715b211460aba9dc69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38191 } }
I20260618 23:35:40.523994 20549 raft_consensus.cc:348] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78fce8a82f304e2caaf12a69f07ba614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46423 } } peers { permanent_uuid: "f6fb60aba59c4524bb710074f0b90809" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40739 } } peers { permanent_uuid: "412f6e1f833a4715b211460aba9dc69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38191 } }
I20260618 23:35:40.524163 20548 ts_tablet_manager.cc:1428] T c3b8a989deb04921b91760396d7f3912 P 412f6e1f833a4715b211460aba9dc69a: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260618 23:35:40.524178 20549 raft_consensus.cc:374] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:40.524313 20534 heartbeater.cc:503] Master 127.0.98.62:42403 was elected leader, sending a full tablet report...
I20260618 23:35:40.524224 20549 raft_consensus.cc:729] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6fb60aba59c4524bb710074f0b90809, State: Initialized, Role: FOLLOWER
I20260618 23:35:40.524780 20549 consensus_queue.cc:260] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78fce8a82f304e2caaf12a69f07ba614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46423 } } peers { permanent_uuid: "f6fb60aba59c4524bb710074f0b90809" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40739 } } peers { permanent_uuid: "412f6e1f833a4715b211460aba9dc69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38191 } }
I20260618 23:35:40.524897 20550 raft_consensus.cc:348] T c3b8a989deb04921b91760396d7f3912 P 78fce8a82f304e2caaf12a69f07ba614 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78fce8a82f304e2caaf12a69f07ba614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46423 } } peers { permanent_uuid: "f6fb60aba59c4524bb710074f0b90809" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40739 } } peers { permanent_uuid: "412f6e1f833a4715b211460aba9dc69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38191 } }
I20260618 23:35:40.525072 20550 raft_consensus.cc:374] T c3b8a989deb04921b91760396d7f3912 P 78fce8a82f304e2caaf12a69f07ba614 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:40.525117 20550 raft_consensus.cc:729] T c3b8a989deb04921b91760396d7f3912 P 78fce8a82f304e2caaf12a69f07ba614 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78fce8a82f304e2caaf12a69f07ba614, State: Initialized, Role: FOLLOWER
I20260618 23:35:40.525130 20549 ts_tablet_manager.cc:1428] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:35:40.525219 20550 consensus_queue.cc:260] T c3b8a989deb04921b91760396d7f3912 P 78fce8a82f304e2caaf12a69f07ba614 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78fce8a82f304e2caaf12a69f07ba614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46423 } } peers { permanent_uuid: "f6fb60aba59c4524bb710074f0b90809" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40739 } } peers { permanent_uuid: "412f6e1f833a4715b211460aba9dc69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38191 } }
I20260618 23:35:40.525264 20272 heartbeater.cc:503] Master 127.0.98.62:42403 was elected leader, sending a full tablet report...
I20260618 23:35:40.525481 20550 ts_tablet_manager.cc:1428] T c3b8a989deb04921b91760396d7f3912 P 78fce8a82f304e2caaf12a69f07ba614: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260618 23:35:40.525556 20403 heartbeater.cc:503] Master 127.0.98.62:42403 was elected leader, sending a full tablet report...
I20260618 23:35:40.549247 20555 raft_consensus.cc:482] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:35:40.549433 20555 raft_consensus.cc:504] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78fce8a82f304e2caaf12a69f07ba614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46423 } } peers { permanent_uuid: "f6fb60aba59c4524bb710074f0b90809" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40739 } } peers { permanent_uuid: "412f6e1f833a4715b211460aba9dc69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38191 } }
I20260618 23:35:40.549902 20555 leader_election.cc:288] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78fce8a82f304e2caaf12a69f07ba614 (127.0.98.2:46423), 412f6e1f833a4715b211460aba9dc69a (127.0.98.3:38191)
I20260618 23:35:40.553503 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:40.550340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37281 (local address 127.0.98.2:46423)
0618 23:35:40.550456 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:40.550459 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:40.550472 (+ 13us) server_negotiation.cc:407] Connection header received
0618 23:35:40.550589 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:40.550590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:40.550609 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:40.550691 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:40.550935 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.551497 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.552673 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.553075 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.553188 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:40.553201 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:40.553360 (+ 159us) server_negotiation.cc:299] Negotiation successful
0618 23:35:40.553411 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260618 23:35:40.553519 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:40.550580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33563 (local address 127.0.98.3:38191)
0618 23:35:40.550649 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:40.550651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:40.550689 (+ 38us) server_negotiation.cc:407] Connection header received
0618 23:35:40.550726 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:40.550727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:40.550738 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:40.550826 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:35:40.551084 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.551623 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.552707 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.553078 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.553186 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:40.553197 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:40.553410 (+ 213us) server_negotiation.cc:299] Negotiation successful
0618 23:35:40.553449 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 23:35:40.553519 20274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:40.550212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46423 (local address 127.0.98.1:37281)
0618 23:35:40.550367 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:40.550390 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:40.550476 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:40.550734 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:40.550736 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:40.550739 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:40.550828 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:40.550830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.551630 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.551632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:40.552546 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:40.552549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.553188 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.553189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:40.553268 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:40.553272 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:35:40.553330 (+ 58us) client_negotiation.cc:240] Negotiation successful
0618 23:35:40.553407 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":7}
I20260618 23:35:40.553536 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:40.550385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38191 (local address 127.0.98.1:33563)
0618 23:35:40.550590 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:40.550600 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:35:40.550655 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:35:40.550843 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:35:40.550844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:35:40.550846 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:35:40.550966 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:40.550969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.551727 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.551727 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:40.552562 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:35:40.552564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.553195 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.553196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:35:40.553288 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:40.553292 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:35:40.553381 (+ 89us) client_negotiation.cc:240] Negotiation successful
0618 23:35:40.553424 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":3,"thread_start_us":122,"threads_started":1}
I20260618 23:35:40.554030 20357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b8a989deb04921b91760396d7f3912" candidate_uuid: "f6fb60aba59c4524bb710074f0b90809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78fce8a82f304e2caaf12a69f07ba614" is_pre_election: true
I20260618 23:35:40.554242 20357 raft_consensus.cc:2454] T c3b8a989deb04921b91760396d7f3912 P 78fce8a82f304e2caaf12a69f07ba614 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6fb60aba59c4524bb710074f0b90809 in term 0.
I20260618 23:35:40.554237 20488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b8a989deb04921b91760396d7f3912" candidate_uuid: "f6fb60aba59c4524bb710074f0b90809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412f6e1f833a4715b211460aba9dc69a" is_pre_election: true
I20260618 23:35:40.554452 20488 raft_consensus.cc:2454] T c3b8a989deb04921b91760396d7f3912 P 412f6e1f833a4715b211460aba9dc69a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6fb60aba59c4524bb710074f0b90809 in term 0.
I20260618 23:35:40.554584 20160 leader_election.cc:302] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78fce8a82f304e2caaf12a69f07ba614, f6fb60aba59c4524bb710074f0b90809; no voters:
I20260618 23:35:40.554831 20555 raft_consensus.cc:2781] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:35:40.554890 20555 raft_consensus.cc:482] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:35:40.554909 20555 raft_consensus.cc:3037] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:40.555919 20555 raft_consensus.cc:504] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78fce8a82f304e2caaf12a69f07ba614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46423 } } peers { permanent_uuid: "f6fb60aba59c4524bb710074f0b90809" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40739 } } peers { permanent_uuid: "412f6e1f833a4715b211460aba9dc69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38191 } }
I20260618 23:35:40.556094 20555 leader_election.cc:288] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809 [CANDIDATE]: Term 1 election: Requested vote from peers 78fce8a82f304e2caaf12a69f07ba614 (127.0.98.2:46423), 412f6e1f833a4715b211460aba9dc69a (127.0.98.3:38191)
I20260618 23:35:40.556404 20357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b8a989deb04921b91760396d7f3912" candidate_uuid: "f6fb60aba59c4524bb710074f0b90809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78fce8a82f304e2caaf12a69f07ba614"
I20260618 23:35:40.556407 20488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3b8a989deb04921b91760396d7f3912" candidate_uuid: "f6fb60aba59c4524bb710074f0b90809" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "412f6e1f833a4715b211460aba9dc69a"
I20260618 23:35:40.556520 20488 raft_consensus.cc:3037] T c3b8a989deb04921b91760396d7f3912 P 412f6e1f833a4715b211460aba9dc69a [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:40.556519 20357 raft_consensus.cc:3037] T c3b8a989deb04921b91760396d7f3912 P 78fce8a82f304e2caaf12a69f07ba614 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:40.557839 20488 raft_consensus.cc:2454] T c3b8a989deb04921b91760396d7f3912 P 412f6e1f833a4715b211460aba9dc69a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6fb60aba59c4524bb710074f0b90809 in term 1.
I20260618 23:35:40.557839 20357 raft_consensus.cc:2454] T c3b8a989deb04921b91760396d7f3912 P 78fce8a82f304e2caaf12a69f07ba614 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6fb60aba59c4524bb710074f0b90809 in term 1.
I20260618 23:35:40.558120 20160 leader_election.cc:302] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78fce8a82f304e2caaf12a69f07ba614, f6fb60aba59c4524bb710074f0b90809; no voters:
I20260618 23:35:40.558246 20555 raft_consensus.cc:2781] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:40.558466 20555 raft_consensus.cc:686] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809 [term 1 LEADER]: Becoming Leader. State: Replica: f6fb60aba59c4524bb710074f0b90809, State: Running, Role: LEADER
I20260618 23:35:40.558621 20555 consensus_queue.cc:237] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78fce8a82f304e2caaf12a69f07ba614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46423 } } peers { permanent_uuid: "f6fb60aba59c4524bb710074f0b90809" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40739 } } peers { permanent_uuid: "412f6e1f833a4715b211460aba9dc69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38191 } }
I20260618 23:35:40.559767 20082 catalog_manager.cc:5495] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6fb60aba59c4524bb710074f0b90809 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f6fb60aba59c4524bb710074f0b90809" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78fce8a82f304e2caaf12a69f07ba614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46423 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6fb60aba59c4524bb710074f0b90809" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40739 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "412f6e1f833a4715b211460aba9dc69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38191 } health_report { overall_health: UNKNOWN } } }
I20260618 23:35:40.574555 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:40.571944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53822 (local address 127.0.98.1:40739)
0618 23:35:40.572019 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:40.572021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:40.572032 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:35:40.572080 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:40.572080 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:40.572103 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:40.572201 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:35:40.572361 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.572889 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.573696 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:40.573918 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:40.573998 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:40.574163 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:35:40.574290 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:35:40.574365 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:40.574426 (+ 61us) server_negotiation.cc:299] Negotiation successful
0618 23:35:40.574450 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260618 23:35:40.575300 20404 tablet.cc:2369] T c3b8a989deb04921b91760396d7f3912 P 78fce8a82f304e2caaf12a69f07ba614: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:35:40.577410 20357 raft_consensus.cc:1261] T c3b8a989deb04921b91760396d7f3912 P 78fce8a82f304e2caaf12a69f07ba614 [term 1 FOLLOWER]: Refusing update from remote peer f6fb60aba59c4524bb710074f0b90809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:35:40.577410 20488 raft_consensus.cc:1261] T c3b8a989deb04921b91760396d7f3912 P 412f6e1f833a4715b211460aba9dc69a [term 1 FOLLOWER]: Refusing update from remote peer f6fb60aba59c4524bb710074f0b90809: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:35:40.577998 20558 consensus_queue.cc:1035] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "412f6e1f833a4715b211460aba9dc69a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:35:40.578177 20555 consensus_queue.cc:1035] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78fce8a82f304e2caaf12a69f07ba614" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:35:40.581317 20563 mvcc.cc:205] Tried to move back new op lower bound from 7298358233400872960 to 7298358233328914432. Current Snapshot: MvccSnapshot[applied={T|T < 7298358233400872960}]
I20260618 23:35:40.583613 20562 mvcc.cc:205] Tried to move back new op lower bound from 7298358233400872960 to 7298358233328914432. Current Snapshot: MvccSnapshot[applied={T|T < 7298358233400872960}]
I20260618 23:35:40.583848 20084 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:35:40.583962 20084 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:35:40.584893 20084 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bcf928a769db4e998fb33d66b4ac9d2c: Sending DeleteTablet for 3 replicas of tablet c3b8a989deb04921b91760396d7f3912
I20260618 23:35:40.585264 20337 tablet_service.cc:1514] Processing DeleteTablet for tablet c3b8a989deb04921b91760396d7f3912 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:35:40 UTC) from {username='slave'} at 127.0.0.1:46654
I20260618 23:35:40.585315 20468 tablet_service.cc:1514] Processing DeleteTablet for tablet c3b8a989deb04921b91760396d7f3912 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:35:40 UTC) from {username='slave'} at 127.0.0.1:50078
I20260618 23:35:40.585546 20206 tablet_service.cc:1514] Processing DeleteTablet for tablet c3b8a989deb04921b91760396d7f3912 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:35:40 UTC) from {username='slave'} at 127.0.0.1:53810
I20260618 23:35:40.585794 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 20144
I20260618 23:35:40.585817 20574 tablet_replica.cc:330] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809: stopping tablet replica
I20260618 23:35:40.586001 20574 raft_consensus.cc:2229] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:35:40.586207 20574 raft_consensus.cc:2258] T c3b8a989deb04921b91760396d7f3912 P f6fb60aba59c4524bb710074f0b90809 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:35:40.591763 20573 tablet_replica.cc:330] T c3b8a989deb04921b91760396d7f3912 P 412f6e1f833a4715b211460aba9dc69a: stopping tablet replica
I20260618 23:35:40.591953 20573 raft_consensus.cc:2229] T c3b8a989deb04921b91760396d7f3912 P 412f6e1f833a4715b211460aba9dc69a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:35:40.591990 20572 tablet_replica.cc:330] T c3b8a989deb04921b91760396d7f3912 P 78fce8a82f304e2caaf12a69f07ba614: stopping tablet replica
I20260618 23:35:40.592087 20572 raft_consensus.cc:2229] T c3b8a989deb04921b91760396d7f3912 P 78fce8a82f304e2caaf12a69f07ba614 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:35:40.592098 20573 raft_consensus.cc:2258] T c3b8a989deb04921b91760396d7f3912 P 412f6e1f833a4715b211460aba9dc69a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:35:40.592185 20572 raft_consensus.cc:2258] T c3b8a989deb04921b91760396d7f3912 P 78fce8a82f304e2caaf12a69f07ba614 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:35:40.592701 20572 ts_tablet_manager.cc:1905] T c3b8a989deb04921b91760396d7f3912 P 78fce8a82f304e2caaf12a69f07ba614: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:35:40.592937 20573 ts_tablet_manager.cc:1905] T c3b8a989deb04921b91760396d7f3912 P 412f6e1f833a4715b211460aba9dc69a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260618 23:35:40.593103 20070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40739 (error 108)
I20260618 23:35:40.593331 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 20275
I20260618 23:35:40.595628 20573 ts_tablet_manager.cc:1918] T c3b8a989deb04921b91760396d7f3912 P 412f6e1f833a4715b211460aba9dc69a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:35:40.595742 20573 log.cc:1196] T c3b8a989deb04921b91760396d7f3912 P 412f6e1f833a4715b211460aba9dc69a: Deleting WAL directory at /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825489608381-392-0/minicluster-data/ts-2/wal/wals/c3b8a989deb04921b91760396d7f3912
I20260618 23:35:40.595803 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:40.595556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40739 (local address 127.0.0.1:53830)
0618 23:35:40.595656 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:40.595740 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40739: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260618 23:35:40.596029 20070 catalog_manager.cc:4584] TS f6fb60aba59c4524bb710074f0b90809 (127.0.98.1:40739): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c3b8a989deb04921b91760396d7f3912: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40739: connect: Connection refused (error 111)
I20260618 23:35:40.596063 20573 ts_tablet_manager.cc:1939] T c3b8a989deb04921b91760396d7f3912 P 412f6e1f833a4715b211460aba9dc69a: Deleting consensus metadata
I20260618 23:35:40.596550 20069 catalog_manager.cc:4841] TS 412f6e1f833a4715b211460aba9dc69a (127.0.98.3:38191): tablet c3b8a989deb04921b91760396d7f3912 (table test-table [id=101f702c1f14468687410285c4badea3]) successfully deleted
I20260618 23:35:40.599931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 20406
I20260618 23:35:40.599964 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:40.599789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46423 (local address 127.0.0.1:46668)
0618 23:35:40.599867 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0618 23:35:40.599916 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260618 23:35:40.605615 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 20053
2026-06-18T23:35:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (740 ms)
[----------] 6 tests from EncryptionPolicyTest (8929 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:35:40 dist-test-slave-6224 krb5kdc[20578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:35:40 dist-test-slave-6224 krb5kdc[20578](info): set up 2 sockets
Jun 18 23:35:40 dist-test-slave-6224 krb5kdc[20578](info): commencing operation
krb5kdc: starting...
W20260618 23:35:41.655817 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:35:41 dist-test-slave-6224 krb5kdc[20578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:35:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:35:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:35:41.783393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38205
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:41.874927 20591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:41.875252 20591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:41.875296 20591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:41.876755 20591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:35:41.876853 20591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:41.876883 20591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:41.876909 20591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:35:41.876957 20591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:35:41.878777 20591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38205
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:41.879134 20591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:41.879412 20591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:41.885423 20597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:41.885433 20599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:41.885433 20596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:41.885617 20591 server_base.cc:1034] running on GCE node
I20260618 23:35:41.886029 20591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:41.886389 20591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:41.887614 20591 hybrid_clock.cc:648] HybridClock initialized: now 1781825741887583 us; error 54 us; skew 500 ppm
Jun 18 23:35:41 dist-test-slave-6224 krb5kdc[20578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:41.893639 20591 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:35:41.894347 20591 webserver.cc:460] Webserver started at http://127.0.0.1:46797/ using document root <none> and password file <none>
I20260618 23:35:41.894644 20591 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:41.894716 20591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:41.894878 20591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:41.896216 20591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "e7a5183841074584a10b57a8cb309285"
format_stamp: "Formatted at 2026-06-18 23:35:41 on dist-test-slave-6224"
server_key: "c000d6d74b135fcb93fa421ef522c731"
server_key_iv: "8f4349b24b7a01137d130328d78cf577"
server_key_version: "encryptionkey@0"
I20260618 23:35:41.896612 20591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "e7a5183841074584a10b57a8cb309285"
format_stamp: "Formatted at 2026-06-18 23:35:41 on dist-test-slave-6224"
server_key: "c000d6d74b135fcb93fa421ef522c731"
server_key_iv: "8f4349b24b7a01137d130328d78cf577"
server_key_version: "encryptionkey@0"
I20260618 23:35:41.898234 20591 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:35:41.899194 20606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:41.899672 20591 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:41.899806 20591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "e7a5183841074584a10b57a8cb309285"
format_stamp: "Formatted at 2026-06-18 23:35:41 on dist-test-slave-6224"
server_key: "c000d6d74b135fcb93fa421ef522c731"
server_key_iv: "8f4349b24b7a01137d130328d78cf577"
server_key_version: "encryptionkey@0"
I20260618 23:35:41.899899 20591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:41.920985 20591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:41.923545 20591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:41.923794 20591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:41.929519 20591 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38205
I20260618 23:35:41.929577 20658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38205 every 8 connection(s)
I20260618 23:35:41.930024 20591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:35:41.931125 20659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:41.933583 20659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285: Bootstrap starting.
I20260618 23:35:41.934178 20659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:41.934489 20659 log.cc:824] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:41.935390 20659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285: No bootstrap required, opened a new log
I20260618 23:35:41.937031 20659 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5183841074584a10b57a8cb309285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38205 } }
I20260618 23:35:41.937181 20659 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:41.937206 20659 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7a5183841074584a10b57a8cb309285, State: Initialized, Role: FOLLOWER
I20260618 23:35:41.937373 20659 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5183841074584a10b57a8cb309285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38205 } }
I20260618 23:35:41.937441 20659 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:35:41.937484 20659 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:35:41.937546 20659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:41.938328 20659 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5183841074584a10b57a8cb309285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38205 } }
I20260618 23:35:41.938431 20659 leader_election.cc:302] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7a5183841074584a10b57a8cb309285; no voters:
I20260618 23:35:41.938664 20659 leader_election.cc:288] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:35:41.938778 20662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:41.938969 20659 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:35:41.939024 20662 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285 [term 1 LEADER]: Becoming Leader. State: Replica: e7a5183841074584a10b57a8cb309285, State: Running, Role: LEADER
I20260618 23:35:41.939162 20662 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5183841074584a10b57a8cb309285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38205 } }
I20260618 23:35:41.939282 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 20591
I20260618 23:35:41.939415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:35:41.939738 392 external_mini_cluster.cc:1428] Setting key ea2afcfd613975e1b9d06834df08ed1b
I20260618 23:35:41.940004 20664 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7a5183841074584a10b57a8cb309285. Latest consensus state: current_term: 1 leader_uuid: "e7a5183841074584a10b57a8cb309285" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5183841074584a10b57a8cb309285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38205 } } }
I20260618 23:35:41.940165 20664 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:41.940582 20663 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7a5183841074584a10b57a8cb309285" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7a5183841074584a10b57a8cb309285" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38205 } } }
I20260618 23:35:41.940665 20663 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285 [sys.catalog]: This master's current role is: LEADER
W20260618 23:35:41.941093 20678 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:35:41.941169 20678 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:35:41.941299 20674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:35:41.941545 20674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:35:41.943657 20674 catalog_manager.cc:1347] Generated new cluster ID: b445b4e0e3164fe39a4ba37100b75c60
I20260618 23:35:41.943723 20674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 18 23:35:41 dist-test-slave-6224 krb5kdc[20578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260618 23:35:41.953838 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:41.940636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60599 (local address 127.0.0.1:38205)
0618 23:35:41.941004 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:41.941047 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:41.941241 (+ 194us) server_negotiation.cc:407] Connection header received
0618 23:35:41.941507 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:41.941515 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:41.941568 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:41.941768 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:41.942513 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:41.944056 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:41.945193 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:41.945925 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:41.946065 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:41.949521 (+ 3456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:41.949527 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:41.949527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:41.949543 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:41.951924 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:41.952480 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:41.952482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:41.952483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:41.952609 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:41.952863 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:41.952864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:41.952865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:41.953180 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:41.953351 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:41.953484 (+ 133us) server_negotiation.cc:299] Negotiation successful
0618 23:35:41.953594 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":84,"threads_started":1}
I20260618 23:35:41.955041 20674 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:35:41.955294 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:35:41.955731 20674 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:35:41.964437 20679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:41.956962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53203 (local address 127.0.0.1:38205)
0618 23:35:41.956999 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:41.957000 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:41.957175 (+ 175us) server_negotiation.cc:407] Connection header received
0618 23:35:41.957259 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:41.957259 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:41.957269 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:41.957296 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:41.958004 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:41.958781 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:41.959804 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:41.960101 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:41.960138 (+ 37us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:41.961639 (+ 1501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:41.961641 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:41.961641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:41.961647 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:41.963191 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:41.963608 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:41.963609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:41.963609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:41.963658 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:41.963875 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:41.963875 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:41.963876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:41.963983 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:41.964067 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:41.964276 (+ 209us) server_negotiation.cc:299] Negotiation successful
0618 23:35:41.964339 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260618 23:35:41.965410 20674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7a5183841074584a10b57a8cb309285: Generated new TSK 0
I20260618 23:35:41.965570 20674 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:35:41.969718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 20591
W20260618 23:35:41.975216 20576 connection.cc:535] client connection to 127.0.0.1:38205 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38205): Connection reset by peer (error 104)
2026-06-18T23:35:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:35:42 dist-test-slave-6224 krb5kdc[20690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:35:42 dist-test-slave-6224 krb5kdc[20690](info): set up 2 sockets
Jun 18 23:35:42 dist-test-slave-6224 krb5kdc[20690](info): commencing operation
krb5kdc: starting...
W20260618 23:35:43.011830 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:35:43 dist-test-slave-6224 krb5kdc[20690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:35:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:35:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:35:43.142536 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:43.227730 20703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:43.227957 20703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:43.227980 20703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:43.229048 20703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:35:43.229087 20703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:43.229102 20703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:43.229115 20703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:35:43.229127 20703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:35:43.230515 20703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35691
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:43.230707 20703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:43.230885 20703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:43.236335 20709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:43.236378 20711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:43.236378 20708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:43.236701 20703 server_base.cc:1034] running on GCE node
I20260618 23:35:43.236975 20703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:43.237280 20703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:43.238471 20703 hybrid_clock.cc:648] HybridClock initialized: now 1781825743238440 us; error 50 us; skew 500 ppm
Jun 18 23:35:43 dist-test-slave-6224 krb5kdc[20690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:43.241222 20703 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:35:43.241725 20703 webserver.cc:460] Webserver started at http://127.0.0.1:40791/ using document root <none> and password file <none>
I20260618 23:35:43.241961 20703 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:43.242031 20703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:43.242185 20703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:43.243291 20703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "a188d3dc341d4b4aa55e205ceace2846"
format_stamp: "Formatted at 2026-06-18 23:35:43 on dist-test-slave-6224"
server_key: "5629e5e315395281e618789e2f2bc54d"
server_key_iv: "55b3040a92043b2a0ead9c5e4f4f823e"
server_key_version: "encryptionkey@0"
I20260618 23:35:43.243710 20703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "a188d3dc341d4b4aa55e205ceace2846"
format_stamp: "Formatted at 2026-06-18 23:35:43 on dist-test-slave-6224"
server_key: "5629e5e315395281e618789e2f2bc54d"
server_key_iv: "55b3040a92043b2a0ead9c5e4f4f823e"
server_key_version: "encryptionkey@0"
I20260618 23:35:43.245090 20703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:43.245883 20718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:43.246098 20703 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:43.246201 20703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "a188d3dc341d4b4aa55e205ceace2846"
format_stamp: "Formatted at 2026-06-18 23:35:43 on dist-test-slave-6224"
server_key: "5629e5e315395281e618789e2f2bc54d"
server_key_iv: "55b3040a92043b2a0ead9c5e4f4f823e"
server_key_version: "encryptionkey@0"
I20260618 23:35:43.246282 20703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:43.272605 20703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:43.274787 20703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:43.274953 20703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:43.279594 20703 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35691
I20260618 23:35:43.279721 20770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35691 every 8 connection(s)
I20260618 23:35:43.280027 20703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:35:43.281051 20771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:43.283741 20771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846: Bootstrap starting.
I20260618 23:35:43.284396 20771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:43.284672 20771 log.cc:824] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:43.285452 20771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846: No bootstrap required, opened a new log
I20260618 23:35:43.287163 20771 raft_consensus.cc:348] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a188d3dc341d4b4aa55e205ceace2846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } }
I20260618 23:35:43.287326 20771 raft_consensus.cc:374] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:43.287350 20771 raft_consensus.cc:729] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a188d3dc341d4b4aa55e205ceace2846, State: Initialized, Role: FOLLOWER
I20260618 23:35:43.287474 20771 consensus_queue.cc:260] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a188d3dc341d4b4aa55e205ceace2846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } }
I20260618 23:35:43.287534 20771 raft_consensus.cc:388] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:35:43.287649 20771 raft_consensus.cc:482] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:35:43.287704 20771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:43.288343 20771 raft_consensus.cc:504] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a188d3dc341d4b4aa55e205ceace2846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } }
I20260618 23:35:43.288466 20771 leader_election.cc:302] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a188d3dc341d4b4aa55e205ceace2846; no voters:
I20260618 23:35:43.288686 20771 leader_election.cc:288] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:35:43.288766 20774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:43.288813 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 20703
I20260618 23:35:43.288972 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:35:43.288987 20774 raft_consensus.cc:686] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846 [term 1 LEADER]: Becoming Leader. State: Replica: a188d3dc341d4b4aa55e205ceace2846, State: Running, Role: LEADER
I20260618 23:35:43.288997 20771 sys_catalog.cc:564] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:35:43.289219 392 external_mini_cluster.cc:1428] Setting key 7c03cfc93f1378abcc3252b40501ef67
I20260618 23:35:43.289162 20774 consensus_queue.cc:237] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a188d3dc341d4b4aa55e205ceace2846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } }
I20260618 23:35:43.289911 20774 sys_catalog.cc:455] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a188d3dc341d4b4aa55e205ceace2846. Latest consensus state: current_term: 1 leader_uuid: "a188d3dc341d4b4aa55e205ceace2846" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a188d3dc341d4b4aa55e205ceace2846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } } }
I20260618 23:35:43.290010 20774 sys_catalog.cc:458] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:43.290372 20775 sys_catalog.cc:455] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a188d3dc341d4b4aa55e205ceace2846" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a188d3dc341d4b4aa55e205ceace2846" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35691 } } }
I20260618 23:35:43.290524 20775 sys_catalog.cc:458] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846 [sys.catalog]: This master's current role is: LEADER
W20260618 23:35:43.291205 20791 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:35:43.291263 20791 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:35:43.291337 20785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:35:43.291522 20785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:35:43.294677 20785 catalog_manager.cc:1347] Generated new cluster ID: 113afcf6e52c47c0b63fac137162c5ae
I20260618 23:35:43.294746 20785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:35:43.295666 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:43.289893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58221 (local address 127.0.0.1:35691)
0618 23:35:43.290212 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:43.290227 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:43.290284 (+ 57us) server_negotiation.cc:407] Connection header received
0618 23:35:43.290446 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:43.290454 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:43.290542 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:43.291121 (+ 579us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:43.291243 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:43.292391 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:43.293323 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:43.293802 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:43.293931 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:43.294821 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:43.294825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:43.294826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:43.295079 (+ 253us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:43.295124 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:43.295139 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:43.295227 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:43.295382 (+ 155us) server_negotiation.cc:299] Negotiation successful
0618 23:35:43.295413 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":76,"threads_started":1}
I20260618 23:35:43.300738 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:35:43.308054 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:43.303920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43013 (local address 127.0.0.1:35691)
0618 23:35:43.303990 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:43.303992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:43.304112 (+ 120us) server_negotiation.cc:407] Connection header received
0618 23:35:43.304197 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:43.304198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:43.304210 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:43.304285 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:43.304478 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:43.305305 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:43.306493 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:43.306807 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:43.306873 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:43.307750 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:43.307751 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:43.307752 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:43.307757 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:43.307775 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:43.307780 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:43.307838 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:43.307951 (+ 113us) server_negotiation.cc:299] Negotiation successful
0618 23:35:43.307967 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 23:35:43.318001 20785 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:35:43.318868 20785 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:35:43.330106 20785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a188d3dc341d4b4aa55e205ceace2846: Generated new TSK 0
I20260618 23:35:43.330405 20785 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:35:43.339421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 20703
W20260618 23:35:43.345235 20688 connection.cc:535] client connection to 127.0.0.1:35691 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35691): Connection reset by peer (error 104)
2026-06-18T23:35:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1370 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:35:43 dist-test-slave-6224 krb5kdc[20802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:35:43 dist-test-slave-6224 krb5kdc[20802](info): set up 2 sockets
Jun 18 23:35:43 dist-test-slave-6224 krb5kdc[20802](info): commencing operation
krb5kdc: starting...
W20260618 23:35:44.387795 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:35:44 dist-test-slave-6224 krb5kdc[20802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825744, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:35:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:35:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:35:44.515103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33411
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:44.605732 20815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:44.605978 20815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:44.606004 20815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:44.607261 20815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:35:44.607311 20815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:44.607327 20815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:44.607340 20815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:35:44.607394 20815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:35:44.608930 20815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33411
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:44.609217 20815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:44.609468 20815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:44.614769 20821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:44.614795 20823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:44.614742 20820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:44.615161 20815 server_base.cc:1034] running on GCE node
I20260618 23:35:44.615475 20815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:44.615816 20815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:44.617004 20815 hybrid_clock.cc:648] HybridClock initialized: now 1781825744616974 us; error 48 us; skew 500 ppm
Jun 18 23:35:44 dist-test-slave-6224 krb5kdc[20802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:44.620007 20815 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:35:44.620544 20815 webserver.cc:460] Webserver started at http://127.0.0.1:45339/ using document root <none> and password file <none>
I20260618 23:35:44.620788 20815 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:44.620857 20815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:44.621003 20815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:44.622171 20815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "25dcb69e74b44b66bcf1ef8a5a8f3f54"
format_stamp: "Formatted at 2026-06-18 23:35:44 on dist-test-slave-6224"
server_key: "ab5ef959267b833c29c77b9e0af90bb5"
server_key_iv: "6a7087e09cc68e002866b2ac42af5f34"
server_key_version: "encryptionkey@0"
I20260618 23:35:44.622557 20815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "25dcb69e74b44b66bcf1ef8a5a8f3f54"
format_stamp: "Formatted at 2026-06-18 23:35:44 on dist-test-slave-6224"
server_key: "ab5ef959267b833c29c77b9e0af90bb5"
server_key_iv: "6a7087e09cc68e002866b2ac42af5f34"
server_key_version: "encryptionkey@0"
I20260618 23:35:44.624075 20815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:35:44.624979 20830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:44.625240 20815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:44.625351 20815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "25dcb69e74b44b66bcf1ef8a5a8f3f54"
format_stamp: "Formatted at 2026-06-18 23:35:44 on dist-test-slave-6224"
server_key: "ab5ef959267b833c29c77b9e0af90bb5"
server_key_iv: "6a7087e09cc68e002866b2ac42af5f34"
server_key_version: "encryptionkey@0"
I20260618 23:35:44.625427 20815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:44.629068 20815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:44.631328 20815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:44.631628 20815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:44.636674 20815 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33411
I20260618 23:35:44.636729 20882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33411 every 8 connection(s)
I20260618 23:35:44.637115 20815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:35:44.638299 20883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:44.640480 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 20815
I20260618 23:35:44.640645 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:35:44.640856 392 external_mini_cluster.cc:1428] Setting key 8174d3730c51a91603ed51b420d3219f
I20260618 23:35:44.640858 20883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54: Bootstrap starting.
I20260618 23:35:44.641613 20883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:44.641957 20883 log.cc:824] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:44.642792 20883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54: No bootstrap required, opened a new log
I20260618 23:35:44.643798 20886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:44.641448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44567 (local address 127.0.0.1:33411)
0618 23:35:44.641715 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:44.641725 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:44.641789 (+ 64us) server_negotiation.cc:407] Connection header received
0618 23:35:44.642341 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:44.642349 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:44.642377 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:44.642551 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:44.643294 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:44.643297 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:44.643298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:44.643302 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:44.643348 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:44.643362 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:44.643476 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:44.643509 (+ 33us) server_negotiation.cc:299] Negotiation successful
0618 23:35:44.643577 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":111,"threads_started":1}
I20260618 23:35:44.644982 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:35:44.645371 20883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25dcb69e74b44b66bcf1ef8a5a8f3f54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33411 } }
I20260618 23:35:44.645509 20883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:44.645531 20883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25dcb69e74b44b66bcf1ef8a5a8f3f54, State: Initialized, Role: FOLLOWER
I20260618 23:35:44.645675 20883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25dcb69e74b44b66bcf1ef8a5a8f3f54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33411 } }
I20260618 23:35:44.645761 20883 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:35:44.645785 20883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:35:44.645859 20883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:44.646462 20883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25dcb69e74b44b66bcf1ef8a5a8f3f54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33411 } }
I20260618 23:35:44.646581 20883 leader_election.cc:302] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25dcb69e74b44b66bcf1ef8a5a8f3f54; no voters:
I20260618 23:35:44.646755 20883 leader_election.cc:288] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:35:44.646912 20892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:44.647038 20883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:35:44.647089 20892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54 [term 1 LEADER]: Becoming Leader. State: Replica: 25dcb69e74b44b66bcf1ef8a5a8f3f54, State: Running, Role: LEADER
I20260618 23:35:44.647197 20892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25dcb69e74b44b66bcf1ef8a5a8f3f54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33411 } }
I20260618 23:35:44.647639 20895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25dcb69e74b44b66bcf1ef8a5a8f3f54. Latest consensus state: current_term: 1 leader_uuid: "25dcb69e74b44b66bcf1ef8a5a8f3f54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25dcb69e74b44b66bcf1ef8a5a8f3f54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33411 } } }
I20260618 23:35:44.648082 20895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:44.648247 20894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25dcb69e74b44b66bcf1ef8a5a8f3f54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25dcb69e74b44b66bcf1ef8a5a8f3f54" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33411 } } }
I20260618 23:35:44.648329 20894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:44.648528 20886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:44.646756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34901 (local address 127.0.0.1:33411)
0618 23:35:44.646817 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:44.646819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:44.646975 (+ 156us) server_negotiation.cc:407] Connection header received
0618 23:35:44.647242 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:44.647244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:44.647250 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:44.647284 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:44.648170 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:44.648171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:44.648171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:44.648176 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:44.648192 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:44.648196 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:44.648265 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:44.648426 (+ 161us) server_negotiation.cc:299] Negotiation successful
0618 23:35:44.648442 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260618 23:35:44.648695 20900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:35:44.649274 20900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:35:44.653005 20900 catalog_manager.cc:1347] Generated new cluster ID: ff5c4749228c4966b510e9aa818ec75b
I20260618 23:35:44.653069 20900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:35:44.659487 20900 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:35:44.659698 20900 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:35:44.662053 20900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25dcb69e74b44b66bcf1ef8a5a8f3f54: Generated new TSK 0
I20260618 23:35:44.662238 20900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:35:44.667941 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 20815
2026-06-18T23:35:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:35:44 dist-test-slave-6224 krb5kdc[20914](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:35:44 dist-test-slave-6224 krb5kdc[20914](info): set up 2 sockets
Jun 18 23:35:44 dist-test-slave-6224 krb5kdc[20914](info): commencing operation
krb5kdc: starting...
W20260618 23:35:45.731820 392 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:35:45 dist-test-slave-6224 krb5kdc[20914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:35:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:35:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:35:45.858562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38209
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:45.950779 20927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:45.951081 20927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:45.951118 20927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:45.952415 20927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:35:45.952486 20927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:45.952508 20927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:45.952529 20927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:35:45.952558 20927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:35:45.954092 20927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:45.954399 20927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:45.954653 20927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:45.960101 20933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:45.960101 20932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:45.960312 20935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:45.960532 20927 server_base.cc:1034] running on GCE node
I20260618 23:35:45.960858 20927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:45.961187 20927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:45.962368 20927 hybrid_clock.cc:648] HybridClock initialized: now 1781825745962351 us; error 54 us; skew 500 ppm
Jun 18 23:35:45 dist-test-slave-6224 krb5kdc[20914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825745, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:45.965078 20927 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:35:45.965571 20927 webserver.cc:460] Webserver started at http://127.0.0.1:39921/ using document root <none> and password file <none>
I20260618 23:35:45.965807 20927 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:45.965873 20927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:45.966024 20927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:45.967182 20927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "21b91fa26cdd4a38ab1d281da0c8c720"
format_stamp: "Formatted at 2026-06-18 23:35:45 on dist-test-slave-6224"
server_key: "01d6d389592a1e65c4fb249f4e45f9c8"
server_key_iv: "5c31ffe3554512facae05f068fcf5d28"
server_key_version: "encryptionkey@0"
I20260618 23:35:45.967619 20927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "21b91fa26cdd4a38ab1d281da0c8c720"
format_stamp: "Formatted at 2026-06-18 23:35:45 on dist-test-slave-6224"
server_key: "01d6d389592a1e65c4fb249f4e45f9c8"
server_key_iv: "5c31ffe3554512facae05f068fcf5d28"
server_key_version: "encryptionkey@0"
I20260618 23:35:45.969147 20927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:35:45.970113 20942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:45.970364 20927 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260618 23:35:45.970497 20927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "21b91fa26cdd4a38ab1d281da0c8c720"
format_stamp: "Formatted at 2026-06-18 23:35:45 on dist-test-slave-6224"
server_key: "01d6d389592a1e65c4fb249f4e45f9c8"
server_key_iv: "5c31ffe3554512facae05f068fcf5d28"
server_key_version: "encryptionkey@0"
I20260618 23:35:45.970603 20927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:45.981428 20927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:45.983952 20927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:45.984129 20927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:45.988952 20927 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38209
I20260618 23:35:45.989034 20994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38209 every 8 connection(s)
I20260618 23:35:45.989424 20927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:35:45.990456 20995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:45.992837 20995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720: Bootstrap starting.
I20260618 23:35:45.993396 20995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:45.993701 20995 log.cc:824] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:45.994174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 20927
I20260618 23:35:45.994278 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:35:45.994508 392 external_mini_cluster.cc:1428] Setting key 2bfcf9a37300344feed10eb5646fd3e2
I20260618 23:35:45.994824 20995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720: No bootstrap required, opened a new log
I20260618 23:35:45.997674 20995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b91fa26cdd4a38ab1d281da0c8c720" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38209 } }
I20260618 23:35:45.997869 20995 raft_consensus.cc:374] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:45.997918 20995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b91fa26cdd4a38ab1d281da0c8c720, State: Initialized, Role: FOLLOWER
I20260618 23:35:45.998087 20995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b91fa26cdd4a38ab1d281da0c8c720" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38209 } }
I20260618 23:35:45.998173 20995 raft_consensus.cc:388] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:35:45.998220 20995 raft_consensus.cc:482] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:35:45.998289 20995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:45.998971 20995 raft_consensus.cc:504] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b91fa26cdd4a38ab1d281da0c8c720" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38209 } }
I20260618 23:35:45.999095 20995 leader_election.cc:302] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 21b91fa26cdd4a38ab1d281da0c8c720; no voters:
I20260618 23:35:45.999343 20995 leader_election.cc:288] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:35:45.999429 21000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:45.999671 21000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720 [term 1 LEADER]: Becoming Leader. State: Replica: 21b91fa26cdd4a38ab1d281da0c8c720, State: Running, Role: LEADER
I20260618 23:35:45.999686 20995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:35:45.999805 21000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b91fa26cdd4a38ab1d281da0c8c720" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38209 } }
I20260618 23:35:46.000226 21000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21b91fa26cdd4a38ab1d281da0c8c720. Latest consensus state: current_term: 1 leader_uuid: "21b91fa26cdd4a38ab1d281da0c8c720" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b91fa26cdd4a38ab1d281da0c8c720" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38209 } } }
I20260618 23:35:46.000470 21000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:46.000391 21001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "21b91fa26cdd4a38ab1d281da0c8c720" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "21b91fa26cdd4a38ab1d281da0c8c720" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38209 } } }
I20260618 23:35:46.000905 21001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:46.001106 21009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:35:46.001694 21009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 18 23:35:46 dist-test-slave-6224 krb5kdc[20914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260618 23:35:46.008395 21009 catalog_manager.cc:1347] Generated new cluster ID: 629bc5c646944c02bb25d6a2437c7c42
I20260618 23:35:46.008473 21009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:35:46.008541 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:45.995011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47371 (local address 127.0.0.1:38209)
0618 23:35:45.995185 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:45.995196 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:45.995242 (+ 46us) server_negotiation.cc:407] Connection header received
0618 23:35:45.996507 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:45.996514 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:45.996567 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:45.996687 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:45.997606 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:45.999063 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:46.000356 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:46.001141 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:46.001252 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:46.004388 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:46.004393 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:46.004393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:46.004406 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:46.006347 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:46.006749 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:46.006753 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:46.006753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:46.006855 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:46.007058 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:46.007059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:46.007059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:46.007338 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:46.007536 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:46.007643 (+ 107us) server_negotiation.cc:299] Negotiation successful
0618 23:35:46.007809 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260618 23:35:46.009837 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:35:46.017673 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:46.011428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51847 (local address 127.0.0.1:38209)
0618 23:35:46.011494 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:46.011496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:46.011578 (+ 82us) server_negotiation.cc:407] Connection header received
0618 23:35:46.011671 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:46.011672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:46.011684 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:46.011739 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:46.012283 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:46.012825 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:46.013554 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:46.013777 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:46.013858 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:46.015144 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:46.015146 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:46.015146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:46.015153 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:46.016574 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:46.016918 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:46.016920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:46.016920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:46.016974 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:46.017165 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:46.017166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:46.017167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:46.017277 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:46.017385 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:46.017480 (+ 95us) server_negotiation.cc:299] Negotiation successful
0618 23:35:46.017568 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 23:35:46.025388 21009 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:35:46.026038 21009 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:35:46.034639 21009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 21b91fa26cdd4a38ab1d281da0c8c720: Generated new TSK 0
I20260618 23:35:46.034885 21009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:35:46.044118 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 20927
2026-06-18T23:35:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:35:46 dist-test-slave-6224 krb5kdc[21026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:35:46 dist-test-slave-6224 krb5kdc[21026](info): set up 2 sockets
Jun 18 23:35:46 dist-test-slave-6224 krb5kdc[21026](info): commencing operation
krb5kdc: starting...
W20260618 23:35:47.115953 392 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:35:47 dist-test-slave-6224 krb5kdc[21026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:35:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:35:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:35:47.242820 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38079
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:47.329303 21039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:47.329556 21039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:47.329595 21039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:47.330739 21039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:35:47.330814 21039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:47.330842 21039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:47.330862 21039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:35:47.330897 21039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:35:47.332412 21039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:47.332696 21039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:47.332926 21039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:47.338383 21047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:47.338402 21045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:47.338621 21039 server_base.cc:1034] running on GCE node
W20260618 23:35:47.338402 21044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:47.339100 21039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:47.339416 21039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:47.340600 21039 hybrid_clock.cc:648] HybridClock initialized: now 1781825747340583 us; error 47 us; skew 500 ppm
Jun 18 23:35:47 dist-test-slave-6224 krb5kdc[21026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825747, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:47.346750 21039 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:35:47.347266 21039 webserver.cc:460] Webserver started at http://127.0.0.1:33955/ using document root <none> and password file <none>
I20260618 23:35:47.347581 21039 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:47.347644 21039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:47.347792 21039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:47.349107 21039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "a391ee587971482a91225e5857e328d8"
format_stamp: "Formatted at 2026-06-18 23:35:47 on dist-test-slave-6224"
server_key: "73a3c7ccc32504cc3a004a702db618e8"
server_key_iv: "1b00b85e288235664956d6f13bfd8108"
server_key_version: "encryptionkey@0"
I20260618 23:35:47.349540 21039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "a391ee587971482a91225e5857e328d8"
format_stamp: "Formatted at 2026-06-18 23:35:47 on dist-test-slave-6224"
server_key: "73a3c7ccc32504cc3a004a702db618e8"
server_key_iv: "1b00b85e288235664956d6f13bfd8108"
server_key_version: "encryptionkey@0"
I20260618 23:35:47.351202 21039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:35:47.352216 21054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:47.352486 21039 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260618 23:35:47.352602 21039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "a391ee587971482a91225e5857e328d8"
format_stamp: "Formatted at 2026-06-18 23:35:47 on dist-test-slave-6224"
server_key: "73a3c7ccc32504cc3a004a702db618e8"
server_key_iv: "1b00b85e288235664956d6f13bfd8108"
server_key_version: "encryptionkey@0"
I20260618 23:35:47.352689 21039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:47.373158 21039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:47.375764 21039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:47.375941 21039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:47.380950 21039 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38079
I20260618 23:35:47.381129 21106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38079 every 8 connection(s)
I20260618 23:35:47.381425 21039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:35:47.382462 21107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:47.385159 21107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8: Bootstrap starting.
I20260618 23:35:47.385815 21107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:47.386144 21107 log.cc:824] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:47.386993 21107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8: No bootstrap required, opened a new log
I20260618 23:35:47.388998 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 21039
I20260618 23:35:47.388932 21107 raft_consensus.cc:348] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a391ee587971482a91225e5857e328d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38079 } }
I20260618 23:35:47.389132 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:35:47.389117 21107 raft_consensus.cc:374] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:47.389166 21107 raft_consensus.cc:729] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a391ee587971482a91225e5857e328d8, State: Initialized, Role: FOLLOWER
I20260618 23:35:47.389367 392 external_mini_cluster.cc:1428] Setting key 5989ede6e90f2ee6102a605a079c32c2
I20260618 23:35:47.389377 21107 consensus_queue.cc:260] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a391ee587971482a91225e5857e328d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38079 } }
I20260618 23:35:47.389501 21107 raft_consensus.cc:388] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:35:47.389554 21107 raft_consensus.cc:482] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:35:47.389600 21107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:47.390411 21107 raft_consensus.cc:504] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a391ee587971482a91225e5857e328d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38079 } }
I20260618 23:35:47.390566 21107 leader_election.cc:302] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a391ee587971482a91225e5857e328d8; no voters:
I20260618 23:35:47.390795 21107 leader_election.cc:288] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:35:47.390841 21112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:47.391026 21112 raft_consensus.cc:686] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8 [term 1 LEADER]: Becoming Leader. State: Replica: a391ee587971482a91225e5857e328d8, State: Running, Role: LEADER
I20260618 23:35:47.391109 21107 sys_catalog.cc:564] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:35:47.391192 21112 consensus_queue.cc:237] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a391ee587971482a91225e5857e328d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38079 } }
I20260618 23:35:47.391789 21114 sys_catalog.cc:455] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a391ee587971482a91225e5857e328d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a391ee587971482a91225e5857e328d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38079 } } }
I20260618 23:35:47.391896 21114 sys_catalog.cc:458] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:47.392184 21113 sys_catalog.cc:455] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a391ee587971482a91225e5857e328d8. Latest consensus state: current_term: 1 leader_uuid: "a391ee587971482a91225e5857e328d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a391ee587971482a91225e5857e328d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38079 } } }
I20260618 23:35:47.392300 21113 sys_catalog.cc:458] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:47.392675 21120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:35:47.393340 21120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:35:47.395015 21120 catalog_manager.cc:1347] Generated new cluster ID: 1833319651714ea99e3e75e3759d6028
I20260618 23:35:47.395176 21120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:35:47.397181 21111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:47.390006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60787 (local address 127.0.0.1:38079)
0618 23:35:47.390562 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:47.390573 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:47.390610 (+ 37us) server_negotiation.cc:407] Connection header received
0618 23:35:47.390745 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:47.390752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:47.390842 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:47.391052 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:47.391402 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:47.393072 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:47.394420 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:47.395132 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:47.395683 (+ 551us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:47.396208 (+ 525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:47.396212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:47.396213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:47.396460 (+ 247us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:47.396503 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:47.396520 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:47.396812 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:47.396859 (+ 47us) server_negotiation.cc:299] Negotiation successful
0618 23:35:47.396933 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":399,"threads_started":1}
I20260618 23:35:47.398403 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:35:47.404701 21111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:47.400141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47231 (local address 127.0.0.1:38079)
0618 23:35:47.400187 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:47.400189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:47.400288 (+ 99us) server_negotiation.cc:407] Connection header received
0618 23:35:47.400399 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:47.400400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:47.400412 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:47.400445 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:47.400664 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:47.401896 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:47.403066 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:47.403407 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:47.403453 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:47.404385 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:47.404387 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:47.404387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:47.404404 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:47.404418 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:47.404423 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:47.404456 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:47.404577 (+ 121us) server_negotiation.cc:299] Negotiation successful
0618 23:35:47.404607 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260618 23:35:47.414686 21120 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:35:47.415632 21120 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:35:47.420414 21120 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a391ee587971482a91225e5857e328d8: Generated new TSK 0
I20260618 23:35:47.420666 21120 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:35:47.423693 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 21039
2026-06-18T23:35:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:35:47 dist-test-slave-6224 krb5kdc[21138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:35:47 dist-test-slave-6224 krb5kdc[21138](info): set up 2 sockets
Jun 18 23:35:47 dist-test-slave-6224 krb5kdc[21138](info): commencing operation
krb5kdc: starting...
W20260618 23:35:48.479919 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:35:48 dist-test-slave-6224 krb5kdc[21138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:35:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:35:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:35:48.610901 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38413
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:48.703775 21151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:48.704035 21151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:48.704082 21151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:48.705250 21151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:35:48.705319 21151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:48.705344 21151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:48.705366 21151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:35:48.705394 21151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:35:48.706895 21151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38413
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:48.707194 21151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:48.707451 21151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:48.713045 21159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:48.713025 21157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:48.713001 21156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:48.713312 21151 server_base.cc:1034] running on GCE node
I20260618 23:35:48.713624 21151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:48.713945 21151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:48.715108 21151 hybrid_clock.cc:648] HybridClock initialized: now 1781825748715082 us; error 50 us; skew 500 ppm
Jun 18 23:35:48 dist-test-slave-6224 krb5kdc[21138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825748, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:48.718091 21151 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:35:48.718700 21151 webserver.cc:460] Webserver started at http://127.0.0.1:46589/ using document root <none> and password file <none>
I20260618 23:35:48.718986 21151 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:48.719060 21151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:48.719223 21151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:48.720562 21151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "04ca77c93790478d89376552dd8adc5f"
format_stamp: "Formatted at 2026-06-18 23:35:48 on dist-test-slave-6224"
server_key: "8ac1808730db66c72db59f092d22b52b"
server_key_iv: "9e7b29da05e4f810cd407a98a93790b8"
server_key_version: "encryptionkey@0"
I20260618 23:35:48.720970 21151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "04ca77c93790478d89376552dd8adc5f"
format_stamp: "Formatted at 2026-06-18 23:35:48 on dist-test-slave-6224"
server_key: "8ac1808730db66c72db59f092d22b52b"
server_key_iv: "9e7b29da05e4f810cd407a98a93790b8"
server_key_version: "encryptionkey@0"
I20260618 23:35:48.722546 21151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:35:48.723450 21166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:48.723701 21151 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:35:48.723837 21151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "04ca77c93790478d89376552dd8adc5f"
format_stamp: "Formatted at 2026-06-18 23:35:48 on dist-test-slave-6224"
server_key: "8ac1808730db66c72db59f092d22b52b"
server_key_iv: "9e7b29da05e4f810cd407a98a93790b8"
server_key_version: "encryptionkey@0"
I20260618 23:35:48.723930 21151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:48.727806 21151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:48.730495 21151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:48.730664 21151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:48.735929 21151 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38413
I20260618 23:35:48.736049 21218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38413 every 8 connection(s)
I20260618 23:35:48.736414 21151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:35:48.737547 21219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:48.740113 21219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f: Bootstrap starting.
I20260618 23:35:48.740712 21219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:48.741005 21219 log.cc:824] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:48.741818 21219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f: No bootstrap required, opened a new log
I20260618 23:35:48.743603 21219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ca77c93790478d89376552dd8adc5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38413 } }
I20260618 23:35:48.743755 21219 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:48.743779 21219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04ca77c93790478d89376552dd8adc5f, State: Initialized, Role: FOLLOWER
I20260618 23:35:48.743955 21219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ca77c93790478d89376552dd8adc5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38413 } }
I20260618 23:35:48.744037 21219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:35:48.744072 21219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:35:48.744125 21219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:48.744661 21219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ca77c93790478d89376552dd8adc5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38413 } }
I20260618 23:35:48.744776 21219 leader_election.cc:302] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04ca77c93790478d89376552dd8adc5f; no voters:
I20260618 23:35:48.744961 21219 leader_election.cc:288] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:35:48.745082 21222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:48.745232 21219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:35:48.745333 21222 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f [term 1 LEADER]: Becoming Leader. State: Replica: 04ca77c93790478d89376552dd8adc5f, State: Running, Role: LEADER
I20260618 23:35:48.745458 21222 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ca77c93790478d89376552dd8adc5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38413 } }
I20260618 23:35:48.746241 21224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04ca77c93790478d89376552dd8adc5f. Latest consensus state: current_term: 1 leader_uuid: "04ca77c93790478d89376552dd8adc5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ca77c93790478d89376552dd8adc5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38413 } } }
I20260618 23:35:48.746388 21224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:48.746481 21223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04ca77c93790478d89376552dd8adc5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04ca77c93790478d89376552dd8adc5f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38413 } } }
I20260618 23:35:48.746560 21223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:48.746555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 21151
I20260618 23:35:48.746706 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:35:48.746721 21231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:35:48.746925 392 external_mini_cluster.cc:1428] Setting key a0ebaaad1af14ced079fb52307089f01
I20260618 23:35:48.747340 21231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:35:48.749428 21231 catalog_manager.cc:1347] Generated new cluster ID: 3b7280270df6469db1ea99c9b0259534
I20260618 23:35:48.749505 21231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:35:48.749883 21241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:48.747654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40911 (local address 127.0.0.1:38413)
0618 23:35:48.748004 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:48.748015 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:48.748049 (+ 34us) server_negotiation.cc:407] Connection header received
0618 23:35:48.748195 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:48.748202 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:48.748238 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:48.748393 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:48.749272 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:48.749276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:48.749276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:48.749280 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:48.749331 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:48.749362 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:48.749423 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:48.749636 (+ 213us) server_negotiation.cc:299] Negotiation successful
0618 23:35:48.749690 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":79,"threads_started":1}
I20260618 23:35:48.751057 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:35:48.754354 21241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:48.752924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50863 (local address 127.0.0.1:38413)
0618 23:35:48.752963 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:48.752965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:48.753262 (+ 297us) server_negotiation.cc:407] Connection header received
0618 23:35:48.753303 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:48.753304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:48.753312 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:48.753385 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:48.754099 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:48.754100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:48.754100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:48.754105 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:48.754125 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:48.754131 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:48.754199 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:48.754266 (+ 67us) server_negotiation.cc:299] Negotiation successful
0618 23:35:48.754282 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260618 23:35:48.757588 21231 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:35:48.757756 21231 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:35:48.760772 21231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04ca77c93790478d89376552dd8adc5f: Generated new TSK 0
I20260618 23:35:48.760967 21231 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:35:48.766521 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 21151
2026-06-18T23:35:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1342 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:35:48 dist-test-slave-6224 krb5kdc[21250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:35:48 dist-test-slave-6224 krb5kdc[21250](info): set up 2 sockets
Jun 18 23:35:48 dist-test-slave-6224 krb5kdc[21250](info): commencing operation
krb5kdc: starting...
W20260618 23:35:49.815760 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:35:49 dist-test-slave-6224 krb5kdc[21250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:35:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:35:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:35:49.942591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41093
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:50.034821 21263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:50.035110 21263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:50.035149 21263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:50.036412 21263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:35:50.036482 21263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:50.036509 21263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:50.036538 21263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:35:50.036569 21263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:35:50.038108 21263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:50.038401 21263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:50.038661 21263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:50.043998 21271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:50.043998 21268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:50.044049 21269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:50.044227 21263 server_base.cc:1034] running on GCE node
I20260618 23:35:50.044546 21263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:50.044885 21263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:50.046036 21263 hybrid_clock.cc:648] HybridClock initialized: now 1781825750046023 us; error 45 us; skew 500 ppm
Jun 18 23:35:50 dist-test-slave-6224 krb5kdc[21250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825750, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:50.049029 21263 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:35:50.049654 21263 webserver.cc:460] Webserver started at http://127.0.0.1:46435/ using document root <none> and password file <none>
I20260618 23:35:50.049942 21263 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:50.049994 21263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:50.050186 21263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:50.051682 21263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "c60293edbf594e64a7426b7287ba7f93"
format_stamp: "Formatted at 2026-06-18 23:35:50 on dist-test-slave-6224"
server_key: "1a8263abfa86cf1f5e6314d809c82889"
server_key_iv: "c499baf68663c6fc575ff22ef80d38d5"
server_key_version: "encryptionkey@0"
I20260618 23:35:50.052165 21263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "c60293edbf594e64a7426b7287ba7f93"
format_stamp: "Formatted at 2026-06-18 23:35:50 on dist-test-slave-6224"
server_key: "1a8263abfa86cf1f5e6314d809c82889"
server_key_iv: "c499baf68663c6fc575ff22ef80d38d5"
server_key_version: "encryptionkey@0"
I20260618 23:35:50.053774 21263 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:35:50.054720 21278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:50.054961 21263 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:50.055078 21263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "c60293edbf594e64a7426b7287ba7f93"
format_stamp: "Formatted at 2026-06-18 23:35:50 on dist-test-slave-6224"
server_key: "1a8263abfa86cf1f5e6314d809c82889"
server_key_iv: "c499baf68663c6fc575ff22ef80d38d5"
server_key_version: "encryptionkey@0"
I20260618 23:35:50.055187 21263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:50.078156 21263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:50.081058 21263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:50.081257 21263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:50.086876 21263 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41093
I20260618 23:35:50.086936 21330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41093 every 8 connection(s)
I20260618 23:35:50.087427 21263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:35:50.088474 21331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:50.088728 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 21263
I20260618 23:35:50.088868 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:35:50.089087 392 external_mini_cluster.cc:1428] Setting key 30a84981d0ace53574493ef223e202a3
I20260618 23:35:50.092118 21331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93: Bootstrap starting.
I20260618 23:35:50.092751 21331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:50.093072 21331 log.cc:824] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:50.093869 21331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93: No bootstrap required, opened a new log
I20260618 23:35:50.096424 21331 raft_consensus.cc:348] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60293edbf594e64a7426b7287ba7f93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41093 } }
I20260618 23:35:50.096626 21331 raft_consensus.cc:374] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:50.096669 21331 raft_consensus.cc:729] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c60293edbf594e64a7426b7287ba7f93, State: Initialized, Role: FOLLOWER
I20260618 23:35:50.096909 21331 consensus_queue.cc:260] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60293edbf594e64a7426b7287ba7f93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41093 } }
I20260618 23:35:50.097002 21331 raft_consensus.cc:388] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:35:50.097034 21331 raft_consensus.cc:482] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:35:50.097143 21331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:50.097923 21331 raft_consensus.cc:504] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60293edbf594e64a7426b7287ba7f93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41093 } }
Jun 18 23:35:50 dist-test-slave-6224 krb5kdc[21250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260618 23:35:50.098048 21331 leader_election.cc:302] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c60293edbf594e64a7426b7287ba7f93; no voters:
I20260618 23:35:50.098258 21331 leader_election.cc:288] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:35:50.098301 21336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:50.098431 21336 raft_consensus.cc:686] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93 [term 1 LEADER]: Becoming Leader. State: Replica: c60293edbf594e64a7426b7287ba7f93, State: Running, Role: LEADER
I20260618 23:35:50.098521 21336 consensus_queue.cc:237] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60293edbf594e64a7426b7287ba7f93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41093 } }
I20260618 23:35:50.098570 21331 sys_catalog.cc:564] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:35:50.098884 21336 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c60293edbf594e64a7426b7287ba7f93. Latest consensus state: current_term: 1 leader_uuid: "c60293edbf594e64a7426b7287ba7f93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60293edbf594e64a7426b7287ba7f93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41093 } } }
I20260618 23:35:50.099061 21336 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:50.098910 21338 sys_catalog.cc:455] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c60293edbf594e64a7426b7287ba7f93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c60293edbf594e64a7426b7287ba7f93" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41093 } } }
I20260618 23:35:50.099159 21338 sys_catalog.cc:458] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:50.099714 21343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:35:50.100409 21343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:35:50.103255 21343 catalog_manager.cc:1347] Generated new cluster ID: 2d733acd839e4eee85286a08d461a889
I20260618 23:35:50.103320 21343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:35:50.104756 21334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:50.089886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.124:49619 (local address 127.0.0.1:41093)
0618 23:35:50.090131 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:50.090140 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:50.090174 (+ 34us) server_negotiation.cc:407] Connection header received
0618 23:35:50.090909 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:50.090919 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:50.090983 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:50.091161 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:50.092086 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:50.093540 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:50.094705 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:50.095288 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:50.095413 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:50.098843 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:50.098849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:50.098850 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:50.098867 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:50.102249 (+ 3382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:50.102853 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:50.102857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:50.102858 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:50.102966 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:50.103650 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:50.103651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:50.103652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:50.103981 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:50.104169 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:50.104401 (+ 232us) server_negotiation.cc:299] Negotiation successful
0618 23:35:50.104515 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":99,"threads_started":1}
I20260618 23:35:50.108903 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:35:50.117300 21343 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:35:50.117966 21343 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:35:50.119391 21334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:50.110532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.124:51739 (local address 127.0.0.1:41093)
0618 23:35:50.110646 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:50.110649 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:50.110713 (+ 64us) server_negotiation.cc:407] Connection header received
0618 23:35:50.110796 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:50.110797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:50.110809 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:50.110867 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:50.111622 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:50.112451 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:50.113561 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:50.114156 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:50.114251 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:50.115920 (+ 1669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:50.115921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:50.115922 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:50.115930 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:50.118006 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:50.118473 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:50.118474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:50.118475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:50.118534 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:50.118756 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:50.118757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:50.118757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:50.118897 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:50.118978 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:50.119210 (+ 232us) server_negotiation.cc:299] Negotiation successful
0618 23:35:50.119279 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260618 23:35:50.132803 21343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c60293edbf594e64a7426b7287ba7f93: Generated new TSK 0
I20260618 23:35:50.133112 21343 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:35:50.140610 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 21263
W20260618 23:35:50.146114 21248 connection.cc:535] client connection to 127.0.0.1:41093 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41093): Connection reset by peer (error 104)
2026-06-18T23:35:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:35:50 dist-test-slave-6224 krb5kdc[21362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:35:50 dist-test-slave-6224 krb5kdc[21362](info): set up 2 sockets
Jun 18 23:35:50 dist-test-slave-6224 krb5kdc[21362](info): commencing operation
krb5kdc: starting...
W20260618 23:35:51.175810 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:35:51 dist-test-slave-6224 krb5kdc[21362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:35:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:35:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:35:51.304350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45517
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:51.389036 21375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:51.389251 21375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:51.389273 21375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:51.390456 21375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:35:51.390509 21375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:51.390524 21375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:51.390537 21375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:35:51.390548 21375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:35:51.392066 21375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:51.392308 21375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:51.392542 21375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:51.398032 21383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:51.398011 21380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:51.398185 21381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:51.398207 21375 server_base.cc:1034] running on GCE node
I20260618 23:35:51.398561 21375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:51.398882 21375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:51.400101 21375 hybrid_clock.cc:648] HybridClock initialized: now 1781825751400073 us; error 50 us; skew 500 ppm
Jun 18 23:35:51 dist-test-slave-6224 krb5kdc[21362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825751, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:51.403102 21375 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:35:51.403757 21375 webserver.cc:460] Webserver started at http://127.0.0.1:45687/ using document root <none> and password file <none>
I20260618 23:35:51.404047 21375 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:51.404105 21375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:51.404312 21375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:51.405663 21375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "3ff2a9bce9fb45a9916731e6cf8b9860"
format_stamp: "Formatted at 2026-06-18 23:35:51 on dist-test-slave-6224"
server_key: "6f8b21ad1cd651959257fba2fc4ab356"
server_key_iv: "8efb7bfcbffdbf1f40bbea01aed5f4eb"
server_key_version: "encryptionkey@0"
I20260618 23:35:51.406096 21375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "3ff2a9bce9fb45a9916731e6cf8b9860"
format_stamp: "Formatted at 2026-06-18 23:35:51 on dist-test-slave-6224"
server_key: "6f8b21ad1cd651959257fba2fc4ab356"
server_key_iv: "8efb7bfcbffdbf1f40bbea01aed5f4eb"
server_key_version: "encryptionkey@0"
I20260618 23:35:51.407763 21375 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260618 23:35:51.408658 21390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:51.408892 21375 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:51.408977 21375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "3ff2a9bce9fb45a9916731e6cf8b9860"
format_stamp: "Formatted at 2026-06-18 23:35:51 on dist-test-slave-6224"
server_key: "6f8b21ad1cd651959257fba2fc4ab356"
server_key_iv: "8efb7bfcbffdbf1f40bbea01aed5f4eb"
server_key_version: "encryptionkey@0"
I20260618 23:35:51.409077 21375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:51.421566 21375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:51.423799 21375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:51.423959 21375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:51.428978 21375 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45517
I20260618 23:35:51.429080 21442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45517 every 8 connection(s)
I20260618 23:35:51.429459 21375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:35:51.430034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 21375
I20260618 23:35:51.430186 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:35:51.430423 392 external_mini_cluster.cc:1428] Setting key 45a10b8736fc7bbfb87dd188d660997c
I20260618 23:35:51.430701 21443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:51.434460 21443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860: Bootstrap starting.
I20260618 23:35:51.435153 21443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:51.435465 21443 log.cc:824] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:51.436429 21443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860: No bootstrap required, opened a new log
I20260618 23:35:51.437426 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:51.431091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.124:37985 (local address 127.0.0.1:45517)
0618 23:35:51.431359 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:51.431368 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:51.431407 (+ 39us) server_negotiation.cc:407] Connection header received
0618 23:35:51.432060 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:51.432069 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:51.432147 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:51.432321 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:51.432571 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:51.433997 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:51.434960 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:51.435609 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:51.435741 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:51.436694 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:51.436699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:51.436700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:51.436902 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:51.436944 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:51.436957 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:51.437061 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:51.437181 (+ 120us) server_negotiation.cc:299] Negotiation successful
0618 23:35:51.437217 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":130,"threads_started":1}
I20260618 23:35:51.438339 21443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ff2a9bce9fb45a9916731e6cf8b9860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45517 } }
I20260618 23:35:51.438525 21443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:51.438567 21443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ff2a9bce9fb45a9916731e6cf8b9860, State: Initialized, Role: FOLLOWER
I20260618 23:35:51.438689 21443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ff2a9bce9fb45a9916731e6cf8b9860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45517 } }
I20260618 23:35:51.438776 21443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:35:51.438807 21443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:35:51.438855 21443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:51.439136 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:35:51.439582 21443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ff2a9bce9fb45a9916731e6cf8b9860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45517 } }
I20260618 23:35:51.439697 21443 leader_election.cc:302] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ff2a9bce9fb45a9916731e6cf8b9860; no voters:
I20260618 23:35:51.439903 21443 leader_election.cc:288] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:35:51.439973 21448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:51.440183 21443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:35:51.440246 21448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860 [term 1 LEADER]: Becoming Leader. State: Replica: 3ff2a9bce9fb45a9916731e6cf8b9860, State: Running, Role: LEADER
I20260618 23:35:51.440402 21448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ff2a9bce9fb45a9916731e6cf8b9860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45517 } }
I20260618 23:35:51.440932 21449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ff2a9bce9fb45a9916731e6cf8b9860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ff2a9bce9fb45a9916731e6cf8b9860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45517 } } }
I20260618 23:35:51.441026 21449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:51.441201 21449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ff2a9bce9fb45a9916731e6cf8b9860. Latest consensus state: current_term: 1 leader_uuid: "3ff2a9bce9fb45a9916731e6cf8b9860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ff2a9bce9fb45a9916731e6cf8b9860" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45517 } } }
I20260618 23:35:51.441255 21449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:51.441737 21461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:35:51.442462 21461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:35:51.445046 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:51.441004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.124:44733 (local address 127.0.0.1:45517)
0618 23:35:51.441054 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:51.441056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:51.441315 (+ 259us) server_negotiation.cc:407] Connection header received
0618 23:35:51.441453 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:51.441454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:51.441467 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:51.441501 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:51.441845 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:51.442484 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:51.443522 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:51.443827 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:51.443873 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:51.444738 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:51.444740 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:51.444740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:51.444746 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:51.444766 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:51.444771 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:51.444843 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:51.444963 (+ 120us) server_negotiation.cc:299] Negotiation successful
0618 23:35:51.444977 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260618 23:35:51.448038 21461 catalog_manager.cc:1347] Generated new cluster ID: 57d18e65b77e43079f709681b7064948
I20260618 23:35:51.448105 21461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:35:51.458956 21461 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:35:51.459646 21461 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:35:51.463627 21461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ff2a9bce9fb45a9916731e6cf8b9860: Generated new TSK 0
I20260618 23:35:51.463831 21461 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:35:51.472302 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 21375
W20260618 23:35:51.477046 21360 connection.cc:535] client connection to 127.0.0.1:45517 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45517): Connection reset by peer (error 104)
2026-06-18T23:35:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:35:51 dist-test-slave-6224 krb5kdc[21474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:35:51 dist-test-slave-6224 krb5kdc[21474](info): set up 2 sockets
Jun 18 23:35:51 dist-test-slave-6224 krb5kdc[21474](info): commencing operation
krb5kdc: starting...
W20260618 23:35:52.527770 392 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:35:52 dist-test-slave-6224 krb5kdc[21474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:35:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:35:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:35:52.655190 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:52.749228 21487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:52.749472 21487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:52.749497 21487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:52.750629 21487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:35:52.750669 21487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:52.750684 21487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:52.750697 21487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:35:52.750710 21487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:35:52.752233 21487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:52.752473 21487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:52.752696 21487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:52.758270 21493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:52.758293 21495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:52.758270 21492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:52.758605 21487 server_base.cc:1034] running on GCE node
I20260618 23:35:52.758896 21487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:52.759256 21487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:52.760443 21487 hybrid_clock.cc:648] HybridClock initialized: now 1781825752760423 us; error 36 us; skew 500 ppm
Jun 18 23:35:52 dist-test-slave-6224 krb5kdc[21474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825752, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:52.766170 21487 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:35:52.766664 21487 webserver.cc:460] Webserver started at http://127.0.0.1:44063/ using document root <none> and password file <none>
I20260618 23:35:52.766939 21487 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:52.767014 21487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:52.767175 21487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:52.768342 21487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "6a445e590c494d8589db3c1fbbd94891"
format_stamp: "Formatted at 2026-06-18 23:35:52 on dist-test-slave-6224"
server_key: "edb97473e491a8b3c47de418bec039d0"
server_key_iv: "501f8e6edd361b88afb40f52926e992f"
server_key_version: "encryptionkey@0"
I20260618 23:35:52.768738 21487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "6a445e590c494d8589db3c1fbbd94891"
format_stamp: "Formatted at 2026-06-18 23:35:52 on dist-test-slave-6224"
server_key: "edb97473e491a8b3c47de418bec039d0"
server_key_iv: "501f8e6edd361b88afb40f52926e992f"
server_key_version: "encryptionkey@0"
I20260618 23:35:52.770219 21487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:35:52.771116 21502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:52.771425 21487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:52.771588 21487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "6a445e590c494d8589db3c1fbbd94891"
format_stamp: "Formatted at 2026-06-18 23:35:52 on dist-test-slave-6224"
server_key: "edb97473e491a8b3c47de418bec039d0"
server_key_iv: "501f8e6edd361b88afb40f52926e992f"
server_key_version: "encryptionkey@0"
I20260618 23:35:52.771688 21487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:52.775262 21487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:52.777554 21487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:52.777833 21487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:52.782536 21487 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41257
I20260618 23:35:52.782598 21554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41257 every 8 connection(s)
I20260618 23:35:52.782974 21487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:35:52.784231 21555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:52.786896 21555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891: Bootstrap starting.
I20260618 23:35:52.787611 21555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:52.787915 21555 log.cc:824] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:52.788761 21555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891: No bootstrap required, opened a new log
I20260618 23:35:52.790544 21555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a445e590c494d8589db3c1fbbd94891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41257 } }
I20260618 23:35:52.790709 21555 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:52.790733 21555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a445e590c494d8589db3c1fbbd94891, State: Initialized, Role: FOLLOWER
I20260618 23:35:52.790863 21555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a445e590c494d8589db3c1fbbd94891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41257 } }
I20260618 23:35:52.790988 21555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:35:52.791021 21555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:35:52.791154 21555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:52.791280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 21487
I20260618 23:35:52.791435 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:35:52.791661 392 external_mini_cluster.cc:1428] Setting key c7935e59cebb8299ee57ce3294ea13fa
I20260618 23:35:52.792124 21555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a445e590c494d8589db3c1fbbd94891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41257 } }
I20260618 23:35:52.792307 21555 leader_election.cc:302] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a445e590c494d8589db3c1fbbd94891; no voters:
I20260618 23:35:52.792611 21555 leader_election.cc:288] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:35:52.792654 21560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:52.792780 21560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891 [term 1 LEADER]: Becoming Leader. State: Replica: 6a445e590c494d8589db3c1fbbd94891, State: Running, Role: LEADER
I20260618 23:35:52.792933 21555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:35:52.792933 21560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a445e590c494d8589db3c1fbbd94891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41257 } }
I20260618 23:35:52.793833 21560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a445e590c494d8589db3c1fbbd94891. Latest consensus state: current_term: 1 leader_uuid: "6a445e590c494d8589db3c1fbbd94891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a445e590c494d8589db3c1fbbd94891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41257 } } }
I20260618 23:35:52.793929 21560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:52.794284 21562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a445e590c494d8589db3c1fbbd94891" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a445e590c494d8589db3c1fbbd94891" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41257 } } }
I20260618 23:35:52.794392 21562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:52.794454 21559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:52.792230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.124:48775 (local address 127.0.0.1:41257)
0618 23:35:52.792411 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:52.792428 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:52.792462 (+ 34us) server_negotiation.cc:407] Connection header received
0618 23:35:52.792602 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:52.792608 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:52.792638 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:52.792822 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:52.793657 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:52.793661 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:52.793661 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:52.793665 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:52.793725 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:52.793738 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:52.794098 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:52.794133 (+ 35us) server_negotiation.cc:299] Negotiation successful
0618 23:35:52.794182 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":86,"threads_started":1}
I20260618 23:35:52.795431 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260618 23:35:52.795437 21576 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:35:52.795491 21576 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:35:52.795538 21569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:35:52.795748 21569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:35:52.797600 21569 catalog_manager.cc:1347] Generated new cluster ID: 2cac93877c4f4439a954a3e8adf6463b
I20260618 23:35:52.797659 21569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:35:52.798211 21559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:52.796882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.124:53227 (local address 127.0.0.1:41257)
0618 23:35:52.796921 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:52.796923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:52.797051 (+ 128us) server_negotiation.cc:407] Connection header received
0618 23:35:52.797148 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:52.797149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:52.797158 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:52.797224 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:52.797955 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:52.797957 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:52.797957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:52.797963 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:52.797983 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:52.797989 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:52.798051 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:52.798124 (+ 73us) server_negotiation.cc:299] Negotiation successful
0618 23:35:52.798141 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260618 23:35:52.808808 21569 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:35:52.809015 21569 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:35:52.812355 21569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a445e590c494d8589db3c1fbbd94891: Generated new TSK 0
I20260618 23:35:52.812554 21569 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:35:52.817392 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 21487
2026-06-18T23:35:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1345 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:35:52 dist-test-slave-6224 krb5kdc[21586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:35:52 dist-test-slave-6224 krb5kdc[21586](info): set up 2 sockets
Jun 18 23:35:52 dist-test-slave-6224 krb5kdc[21586](info): commencing operation
krb5kdc: starting...
W20260618 23:35:53.867869 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:35:53 dist-test-slave-6224 krb5kdc[21586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:35:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:35:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:35:53.998265 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:54.082810 21599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:54.083096 21599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:54.083137 21599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:54.084298 21599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:35:54.084380 21599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:54.084407 21599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:54.084434 21599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:35:54.084466 21599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:35:54.085917 21599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33781
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:54.086210 21599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:54.086436 21599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:54.092056 21604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:54.092056 21607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:54.092479 21605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:54.093333 21599 server_base.cc:1034] running on GCE node
I20260618 23:35:54.093695 21599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:54.094134 21599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:54.095309 21599 hybrid_clock.cc:648] HybridClock initialized: now 1781825754095281 us; error 46 us; skew 500 ppm
Jun 18 23:35:54 dist-test-slave-6224 krb5kdc[21586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825754, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:54.098438 21599 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:35:54.098973 21599 webserver.cc:460] Webserver started at http://127.0.0.1:41633/ using document root <none> and password file <none>
I20260618 23:35:54.099186 21599 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:54.099236 21599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:54.099330 21599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:54.100421 21599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "4d6b432254f042ddae93ce3141c1507b"
format_stamp: "Formatted at 2026-06-18 23:35:54 on dist-test-slave-6224"
server_key: "d347bc0908c2dc953189a5efb236aa5d"
server_key_iv: "4963d68766870dbe91db5680345fd934"
server_key_version: "encryptionkey@0"
I20260618 23:35:54.100726 21599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "4d6b432254f042ddae93ce3141c1507b"
format_stamp: "Formatted at 2026-06-18 23:35:54 on dist-test-slave-6224"
server_key: "d347bc0908c2dc953189a5efb236aa5d"
server_key_iv: "4963d68766870dbe91db5680345fd934"
server_key_version: "encryptionkey@0"
I20260618 23:35:54.102077 21599 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260618 23:35:54.102866 21614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:54.103101 21599 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:35:54.103185 21599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "4d6b432254f042ddae93ce3141c1507b"
format_stamp: "Formatted at 2026-06-18 23:35:54 on dist-test-slave-6224"
server_key: "d347bc0908c2dc953189a5efb236aa5d"
server_key_iv: "4963d68766870dbe91db5680345fd934"
server_key_version: "encryptionkey@0"
I20260618 23:35:54.103242 21599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:54.117937 21599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:54.120036 21599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:54.120149 21599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:54.124441 21599 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33781
I20260618 23:35:54.124541 21666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33781 every 8 connection(s)
I20260618 23:35:54.124891 21599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:35:54.125945 21667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:54.128670 21667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b: Bootstrap starting.
I20260618 23:35:54.129361 21667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:54.129685 21667 log.cc:824] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:54.130592 21667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b: No bootstrap required, opened a new log
I20260618 23:35:54.132464 21667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d6b432254f042ddae93ce3141c1507b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33781 } }
I20260618 23:35:54.132632 21667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:54.132659 21667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d6b432254f042ddae93ce3141c1507b, State: Initialized, Role: FOLLOWER
I20260618 23:35:54.132840 21667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d6b432254f042ddae93ce3141c1507b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33781 } }
I20260618 23:35:54.132916 21667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:35:54.132941 21667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:35:54.133008 21667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:54.133672 21667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d6b432254f042ddae93ce3141c1507b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33781 } }
I20260618 23:35:54.133802 21667 leader_election.cc:302] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d6b432254f042ddae93ce3141c1507b; no voters:
I20260618 23:35:54.134035 21667 leader_election.cc:288] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:35:54.134094 21670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:54.134394 21670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b [term 1 LEADER]: Becoming Leader. State: Replica: 4d6b432254f042ddae93ce3141c1507b, State: Running, Role: LEADER
I20260618 23:35:54.134395 21667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:35:54.134531 21670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d6b432254f042ddae93ce3141c1507b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33781 } }
I20260618 23:35:54.135017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 21599
I20260618 23:35:54.135162 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:35:54.135185 21672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d6b432254f042ddae93ce3141c1507b. Latest consensus state: current_term: 1 leader_uuid: "4d6b432254f042ddae93ce3141c1507b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d6b432254f042ddae93ce3141c1507b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33781 } } }
I20260618 23:35:54.135275 21672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:54.135425 392 external_mini_cluster.cc:1428] Setting key f96d962322e8f6bf1ba38fc5981c8077
I20260618 23:35:54.135643 21671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d6b432254f042ddae93ce3141c1507b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d6b432254f042ddae93ce3141c1507b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33781 } } }
I20260618 23:35:54.135727 21671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b [sys.catalog]: This master's current role is: LEADER
W20260618 23:35:54.136766 21687 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:35:54.136833 21687 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:35:54.136904 21679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:35:54.137106 21679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:35:54.139019 21679 catalog_manager.cc:1347] Generated new cluster ID: 8ccb7e65f23c42a9a927142d5de4d019
I20260618 23:35:54.139077 21679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 18 23:35:54 dist-test-slave-6224 krb5kdc[21586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260618 23:35:54.146564 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:54.136046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.124:36969 (local address 127.0.0.1:33781)
0618 23:35:54.136234 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:54.136244 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:54.136280 (+ 36us) server_negotiation.cc:407] Connection header received
0618 23:35:54.136431 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:54.136437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:54.136487 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:54.136642 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:54.137427 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:54.138677 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:54.139636 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:54.140087 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:54.140221 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:54.142782 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:54.142786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:54.142787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:54.142800 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:54.144642 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:54.145040 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:54.145043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:54.145043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:54.145149 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:54.145409 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:54.145410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:54.145410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:54.145854 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:54.146103 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:54.146203 (+ 100us) server_negotiation.cc:299] Negotiation successful
0618 23:35:54.146325 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":87,"threads_started":1}
I20260618 23:35:54.147732 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:35:54.156260 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:54.149308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.124:38021 (local address 127.0.0.1:33781)
0618 23:35:54.149352 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:54.149354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:54.149487 (+ 133us) server_negotiation.cc:407] Connection header received
0618 23:35:54.149570 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:54.149571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:54.149581 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:54.149642 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:54.150208 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:54.150757 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:54.151513 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:54.151817 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:54.151914 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:54.153306 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:54.153307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:54.153308 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:54.153315 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:54.154946 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:54.155347 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:54.155348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:54.155349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:54.155410 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:54.155663 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:54.155664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:54.155664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:54.155779 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:54.155919 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:54.156056 (+ 137us) server_negotiation.cc:299] Negotiation successful
0618 23:35:54.156127 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:35:54.167445 21679 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:35:54.168382 21679 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:35:54.175535 21679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d6b432254f042ddae93ce3141c1507b: Generated new TSK 0
I20260618 23:35:54.175820 21679 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:35:54.182421 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 21599
W20260618 23:35:54.187832 21584 connection.cc:535] client connection to 127.0.0.1:33781 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33781): Connection reset by peer (error 104)
2026-06-18T23:35:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:35:54 dist-test-slave-6224 krb5kdc[21698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:35:54 dist-test-slave-6224 krb5kdc[21698](info): set up 2 sockets
Jun 18 23:35:54 dist-test-slave-6224 krb5kdc[21698](info): commencing operation
krb5kdc: starting...
W20260618 23:35:55.211768 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:35:55 dist-test-slave-6224 krb5kdc[21698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825755, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:35:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:35:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:35:55.339184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:55.429200 21711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:55.429491 21711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:55.429529 21711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:55.430716 21711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:35:55.430791 21711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:55.430816 21711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:55.430841 21711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:35:55.430886 21711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:35:55.432461 21711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42641
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:55.432780 21711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:55.433040 21711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:55.438742 21717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:55.438760 21716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:55.438932 21711 server_base.cc:1034] running on GCE node
W20260618 23:35:55.438746 21719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:55.439358 21711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:55.439831 21711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:55.441015 21711 hybrid_clock.cc:648] HybridClock initialized: now 1781825755440988 us; error 54 us; skew 500 ppm
Jun 18 23:35:55 dist-test-slave-6224 krb5kdc[21698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825755, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:55.444309 21711 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:35:55.444958 21711 webserver.cc:460] Webserver started at http://127.0.0.1:37889/ using document root <none> and password file <none>
I20260618 23:35:55.445253 21711 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:55.445328 21711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:55.445497 21711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:55.446875 21711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "ed59b756e5b24d18b0542d2beac468f5"
format_stamp: "Formatted at 2026-06-18 23:35:55 on dist-test-slave-6224"
server_key: "0a53a456790562e883ff626ade66a7e3"
server_key_iv: "b7327fe1258d6fd9b09fd889ad63bbd3"
server_key_version: "encryptionkey@0"
I20260618 23:35:55.447309 21711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "ed59b756e5b24d18b0542d2beac468f5"
format_stamp: "Formatted at 2026-06-18 23:35:55 on dist-test-slave-6224"
server_key: "0a53a456790562e883ff626ade66a7e3"
server_key_iv: "b7327fe1258d6fd9b09fd889ad63bbd3"
server_key_version: "encryptionkey@0"
I20260618 23:35:55.449083 21711 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260618 23:35:55.450448 21726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:55.450726 21711 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:55.450923 21711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "ed59b756e5b24d18b0542d2beac468f5"
format_stamp: "Formatted at 2026-06-18 23:35:55 on dist-test-slave-6224"
server_key: "0a53a456790562e883ff626ade66a7e3"
server_key_iv: "b7327fe1258d6fd9b09fd889ad63bbd3"
server_key_version: "encryptionkey@0"
I20260618 23:35:55.451040 21711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:55.459368 21711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:55.461848 21711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:55.462041 21711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:55.467078 21711 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42641
I20260618 23:35:55.467152 21778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42641 every 8 connection(s)
I20260618 23:35:55.467730 21711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:35:55.468978 21779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:55.472067 21779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5: Bootstrap starting.
I20260618 23:35:55.472774 21779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:55.473048 21779 log.cc:824] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:55.473949 21779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5: No bootstrap required, opened a new log
I20260618 23:35:55.475260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 21711
I20260618 23:35:55.475409 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:35:55.475649 392 external_mini_cluster.cc:1428] Setting key 20798e7c532f48c2a9d54840f44c8dc9
I20260618 23:35:55.476326 21779 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed59b756e5b24d18b0542d2beac468f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42641 } }
I20260618 23:35:55.476548 21779 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:55.476594 21779 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed59b756e5b24d18b0542d2beac468f5, State: Initialized, Role: FOLLOWER
I20260618 23:35:55.476752 21779 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed59b756e5b24d18b0542d2beac468f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42641 } }
I20260618 23:35:55.476841 21779 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:35:55.476886 21779 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:35:55.476938 21779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:55.478129 21779 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed59b756e5b24d18b0542d2beac468f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42641 } }
I20260618 23:35:55.478266 21779 leader_election.cc:302] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed59b756e5b24d18b0542d2beac468f5; no voters:
I20260618 23:35:55.478463 21779 leader_election.cc:288] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:35:55.478549 21784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:55.478782 21779 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:35:55.478928 21784 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5 [term 1 LEADER]: Becoming Leader. State: Replica: ed59b756e5b24d18b0542d2beac468f5, State: Running, Role: LEADER
I20260618 23:35:55.479110 21784 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed59b756e5b24d18b0542d2beac468f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42641 } }
I20260618 23:35:55.479846 21785 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed59b756e5b24d18b0542d2beac468f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed59b756e5b24d18b0542d2beac468f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42641 } } }
I20260618 23:35:55.479965 21785 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:55.480207 21786 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed59b756e5b24d18b0542d2beac468f5. Latest consensus state: current_term: 1 leader_uuid: "ed59b756e5b24d18b0542d2beac468f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed59b756e5b24d18b0542d2beac468f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42641 } } }
I20260618 23:35:55.480242 21795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:35:55.480289 21786 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:55.480978 21795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:35:55.482463 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:55.476347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.124:57491 (local address 127.0.0.1:42641)
0618 23:35:55.476595 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:55.476623 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:55.476665 (+ 42us) server_negotiation.cc:407] Connection header received
0618 23:35:55.476831 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:55.476838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:55.476920 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:55.477116 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:55.477424 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:55.478604 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:55.479854 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:55.480545 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:55.481053 (+ 508us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:55.481584 (+ 531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:55.481590 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:55.481590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:55.481885 (+ 295us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:55.481926 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:55.481940 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:55.482055 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:55.482169 (+ 114us) server_negotiation.cc:299] Negotiation successful
0618 23:35:55.482206 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":182,"thread_start_us":148,"threads_started":1}
I20260618 23:35:55.483191 21795 catalog_manager.cc:1347] Generated new cluster ID: c5a394e8695840da82e325ea47488b00
I20260618 23:35:55.483258 21795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:35:55.483570 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:35:55.490666 21795 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:35:55.491446 21795 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:35:55.493628 21783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:55.487925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.124:40991 (local address 127.0.0.1:42641)
0618 23:35:55.487974 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:55.487976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:55.490172 (+ 2196us) server_negotiation.cc:407] Connection header received
0618 23:35:55.490218 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:55.490220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:55.490241 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:55.490329 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:55.490556 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:55.491304 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:55.492172 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:55.492403 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:55.492494 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:55.493313 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:55.493315 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:55.493315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:55.493320 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:55.493353 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:55.493358 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:55.493427 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:55.493496 (+ 69us) server_negotiation.cc:299] Negotiation successful
0618 23:35:55.493511 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260618 23:35:55.500752 21795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed59b756e5b24d18b0542d2beac468f5: Generated new TSK 0
I20260618 23:35:55.500970 21795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:35:55.505656 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 21711
W20260618 23:35:55.510918 21696 connection.cc:535] client connection to 127.0.0.1:42641 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42641): Connection reset by peer (error 104)
2026-06-18T23:35:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:35:55 dist-test-slave-6224 krb5kdc[21810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:35:55 dist-test-slave-6224 krb5kdc[21810](info): set up 2 sockets
Jun 18 23:35:55 dist-test-slave-6224 krb5kdc[21810](info): commencing operation
krb5kdc: starting...
W20260618 23:35:56.564316 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:35:56 dist-test-slave-6224 krb5kdc[21810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:35:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:35:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:35:56.699328 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40491
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43777
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:56.785569 21823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:56.785863 21823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:56.785907 21823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:56.787096 21823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:35:56.787175 21823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:56.787200 21823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:56.787230 21823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:35:56.787262 21823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:35:56.788858 21823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43777
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40491
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:56.789166 21823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:56.789407 21823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:56.795176 21829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:56.795280 21828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:56.795176 21831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:56.795531 21823 server_base.cc:1034] running on GCE node
I20260618 23:35:56.795853 21823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:56.796130 21823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:56.797309 21823 hybrid_clock.cc:648] HybridClock initialized: now 1781825756797288 us; error 54 us; skew 500 ppm
Jun 18 23:35:56 dist-test-slave-6224 krb5kdc[21810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825756, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:56.800158 21823 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:35:56.800676 21823 webserver.cc:460] Webserver started at http://127.0.0.1:43379/ using document root <none> and password file <none>
I20260618 23:35:56.800953 21823 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:56.801023 21823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:56.801175 21823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:56.802424 21823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "4ee4a479f1a24e8fb0d519c9dd4eae3a"
format_stamp: "Formatted at 2026-06-18 23:35:56 on dist-test-slave-6224"
server_key: "ec340a35cd64a67c4da29984103491bd"
server_key_iv: "6e643b3dbc10508dc03e4f9af58b3313"
server_key_version: "encryptionkey@0"
I20260618 23:35:56.802837 21823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "4ee4a479f1a24e8fb0d519c9dd4eae3a"
format_stamp: "Formatted at 2026-06-18 23:35:56 on dist-test-slave-6224"
server_key: "ec340a35cd64a67c4da29984103491bd"
server_key_iv: "6e643b3dbc10508dc03e4f9af58b3313"
server_key_version: "encryptionkey@0"
I20260618 23:35:56.804412 21823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:35:56.805276 21838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:56.805612 21823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:56.805712 21823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "4ee4a479f1a24e8fb0d519c9dd4eae3a"
format_stamp: "Formatted at 2026-06-18 23:35:56 on dist-test-slave-6224"
server_key: "ec340a35cd64a67c4da29984103491bd"
server_key_iv: "6e643b3dbc10508dc03e4f9af58b3313"
server_key_version: "encryptionkey@0"
I20260618 23:35:56.805819 21823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:56.809389 21823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:56.811815 21823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:56.811975 21823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:56.816761 21823 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40491
I20260618 23:35:56.816879 21890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40491 every 8 connection(s)
I20260618 23:35:56.817209 21823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:35:56.818441 21891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:56.821429 21891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a: Bootstrap starting.
I20260618 23:35:56.822115 21891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:56.822404 21891 log.cc:824] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:56.823412 21891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a: No bootstrap required, opened a new log
I20260618 23:35:56.825789 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 21823
I20260618 23:35:56.825944 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:35:56.825770 21891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee4a479f1a24e8fb0d519c9dd4eae3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40491 } }
I20260618 23:35:56.826025 21891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:56.826064 21891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee4a479f1a24e8fb0d519c9dd4eae3a, State: Initialized, Role: FOLLOWER
I20260618 23:35:56.826172 392 external_mini_cluster.cc:1428] Setting key c61e201fe74e8c566788b3ae3a1ebb97
I20260618 23:35:56.826478 21891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee4a479f1a24e8fb0d519c9dd4eae3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40491 } }
I20260618 23:35:56.826653 21891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:35:56.826704 21891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:35:56.826792 21891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:56.828284 21891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee4a479f1a24e8fb0d519c9dd4eae3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40491 } }
I20260618 23:35:56.828441 21891 leader_election.cc:302] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ee4a479f1a24e8fb0d519c9dd4eae3a; no voters:
I20260618 23:35:56.828662 21891 leader_election.cc:288] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:35:56.828790 21896 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:56.829023 21896 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a [term 1 LEADER]: Becoming Leader. State: Replica: 4ee4a479f1a24e8fb0d519c9dd4eae3a, State: Running, Role: LEADER
I20260618 23:35:56.829167 21891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:35:56.829198 21896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee4a479f1a24e8fb0d519c9dd4eae3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40491 } }
I20260618 23:35:56.829871 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:56.827162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.124:45369 (local address 127.0.0.1:40491)
0618 23:35:56.827470 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:56.827480 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:56.827517 (+ 37us) server_negotiation.cc:407] Connection header received
0618 23:35:56.827793 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:56.827800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:56.827830 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:56.828013 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:56.829032 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:56.829037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:56.829037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:56.829044 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:56.829118 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:56.829138 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:56.829244 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:56.829526 (+ 282us) server_negotiation.cc:299] Negotiation successful
0618 23:35:56.829588 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":150,"threads_started":1}
I20260618 23:35:56.830344 21898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ee4a479f1a24e8fb0d519c9dd4eae3a. Latest consensus state: current_term: 1 leader_uuid: "4ee4a479f1a24e8fb0d519c9dd4eae3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee4a479f1a24e8fb0d519c9dd4eae3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40491 } } }
I20260618 23:35:56.830456 21897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ee4a479f1a24e8fb0d519c9dd4eae3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ee4a479f1a24e8fb0d519c9dd4eae3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40491 } } }
I20260618 23:35:56.830638 21897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:56.830626 21898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:56.831051 21906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:35:56.831636 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:35:56.831780 21906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:35:56.833887 21906 catalog_manager.cc:1347] Generated new cluster ID: 3d91db4f82e94417a35e355c2dfbd250
I20260618 23:35:56.833972 21906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:35:56.836737 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:56.833410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.124:45157 (local address 127.0.0.1:40491)
0618 23:35:56.833471 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:56.833472 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:56.833574 (+ 102us) server_negotiation.cc:407] Connection header received
0618 23:35:56.833645 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:56.833646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:56.833652 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:56.833709 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:56.836426 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:56.836427 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:56.836427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:56.836432 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:56.836455 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:56.836464 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:56.836532 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:56.836650 (+ 118us) server_negotiation.cc:299] Negotiation successful
0618 23:35:56.836666 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260618 23:35:56.839735 21906 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:35:56.839895 21906 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:35:56.844193 21906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ee4a479f1a24e8fb0d519c9dd4eae3a: Generated new TSK 0
I20260618 23:35:56.844384 21906 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:35:56.847213 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 21823
2026-06-18T23:35:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:35:56 dist-test-slave-6224 krb5kdc[21922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:35:56 dist-test-slave-6224 krb5kdc[21922](info): set up 2 sockets
Jun 18 23:35:56 dist-test-slave-6224 krb5kdc[21922](info): commencing operation
krb5kdc: starting...
W20260618 23:35:57.883949 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:35:57 dist-test-slave-6224 krb5kdc[21922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:35:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:35:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:35:58.014788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46807
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:58.106020 21935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:58.106287 21935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:58.106316 21935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:58.107613 21935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:35:58.107658 21935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:58.107673 21935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:58.107689 21935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:35:58.107744 21935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:35:58.109306 21935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46807
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:58.109603 21935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:58.109863 21935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:58.116472 21940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:58.116524 21941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:58.116472 21943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:58.116927 21935 server_base.cc:1034] running on GCE node
I20260618 23:35:58.117213 21935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:58.117553 21935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:58.118786 21935 hybrid_clock.cc:648] HybridClock initialized: now 1781825758118760 us; error 76 us; skew 500 ppm
Jun 18 23:35:58 dist-test-slave-6224 krb5kdc[21922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825758, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:58.126113 21935 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:35:58.126618 21935 webserver.cc:460] Webserver started at http://127.0.98.62:35783/ using document root <none> and password file <none>
I20260618 23:35:58.126878 21935 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:58.126950 21935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:58.127108 21935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:58.128293 21935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "d4357f8bfd674c2fb3bdfdf14b7fc7d2"
format_stamp: "Formatted at 2026-06-18 23:35:58 on dist-test-slave-6224"
server_key: "c3043c4641fbe0bd215ce3a3afa7b7cc"
server_key_iv: "df3fa7471f8c35c673226668e1160e63"
server_key_version: "encryptionkey@0"
I20260618 23:35:58.128680 21935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "d4357f8bfd674c2fb3bdfdf14b7fc7d2"
format_stamp: "Formatted at 2026-06-18 23:35:58 on dist-test-slave-6224"
server_key: "c3043c4641fbe0bd215ce3a3afa7b7cc"
server_key_iv: "df3fa7471f8c35c673226668e1160e63"
server_key_version: "encryptionkey@0"
I20260618 23:35:58.130141 21935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:35:58.131105 21950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:58.131420 21935 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:35:58.131542 21935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "d4357f8bfd674c2fb3bdfdf14b7fc7d2"
format_stamp: "Formatted at 2026-06-18 23:35:58 on dist-test-slave-6224"
server_key: "c3043c4641fbe0bd215ce3a3afa7b7cc"
server_key_iv: "df3fa7471f8c35c673226668e1160e63"
server_key_version: "encryptionkey@0"
I20260618 23:35:58.131705 21935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:58.142591 21935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:58.145488 21935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:58.145637 21935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:58.151090 21935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46807
I20260618 23:35:58.151433 22002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46807 every 8 connection(s)
I20260618 23:35:58.151687 21935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:35:58.152518 22003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:58.154984 22003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2: Bootstrap starting.
I20260618 23:35:58.155684 22003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:58.155993 22003 log.cc:824] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:58.156915 22003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2: No bootstrap required, opened a new log
I20260618 23:35:58.158555 22003 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4357f8bfd674c2fb3bdfdf14b7fc7d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46807 } }
I20260618 23:35:58.158706 22003 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:58.158730 22003 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4357f8bfd674c2fb3bdfdf14b7fc7d2, State: Initialized, Role: FOLLOWER
I20260618 23:35:58.158895 22003 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4357f8bfd674c2fb3bdfdf14b7fc7d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46807 } }
I20260618 23:35:58.158964 22003 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:35:58.158985 22003 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:35:58.159037 22003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:58.160028 22003 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4357f8bfd674c2fb3bdfdf14b7fc7d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46807 } }
I20260618 23:35:58.160125 22003 leader_election.cc:302] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4357f8bfd674c2fb3bdfdf14b7fc7d2; no voters:
I20260618 23:35:58.160358 22003 leader_election.cc:288] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:35:58.160492 22006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:58.160643 22003 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:35:58.160775 22006 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2 [term 1 LEADER]: Becoming Leader. State: Replica: d4357f8bfd674c2fb3bdfdf14b7fc7d2, State: Running, Role: LEADER
I20260618 23:35:58.160938 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 21935
I20260618 23:35:58.160907 22006 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4357f8bfd674c2fb3bdfdf14b7fc7d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46807 } }
I20260618 23:35:58.161070 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:35:58.161262 392 external_mini_cluster.cc:1428] Setting key e92e166c6bd1ca970b76c989858d9de6
I20260618 23:35:58.161401 22008 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4357f8bfd674c2fb3bdfdf14b7fc7d2. Latest consensus state: current_term: 1 leader_uuid: "d4357f8bfd674c2fb3bdfdf14b7fc7d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4357f8bfd674c2fb3bdfdf14b7fc7d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46807 } } }
I20260618 23:35:58.161423 22007 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4357f8bfd674c2fb3bdfdf14b7fc7d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4357f8bfd674c2fb3bdfdf14b7fc7d2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46807 } } }
I20260618 23:35:58.161602 22008 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:58.161661 22007 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:58.162529 22010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:35:58.163233 22010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:35:58.165199 22010 catalog_manager.cc:1347] Generated new cluster ID: 402e618f1c5e4071ab718927ffd6c1c3
I20260618 23:35:58.165258 22010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 18 23:35:58 dist-test-slave-6224 krb5kdc[21922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825757, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:35:58.173641 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:58.161983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55425 (local address 127.0.98.62:46807)
0618 23:35:58.162200 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:58.162215 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:58.162312 (+ 97us) server_negotiation.cc:407] Connection header received
0618 23:35:58.162534 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:58.162544 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:58.162635 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:58.163826 (+ 1191us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:58.163942 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:58.165294 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:58.166213 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:58.166686 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:58.166818 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:58.169503 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:58.169507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:58.169508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:58.169520 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:58.171941 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:58.172353 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:58.172356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:58.172357 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:58.172477 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:58.172711 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:58.172712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:58.172713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:58.172992 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:58.173194 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:58.173284 (+ 90us) server_negotiation.cc:299] Negotiation successful
0618 23:35:58.173401 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":113,"threads_started":1}
I20260618 23:35:58.174885 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:35:58.175695 22010 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:35:58.176435 22010 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:35:58.185232 22010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4357f8bfd674c2fb3bdfdf14b7fc7d2: Generated new TSK 0
I20260618 23:35:58.185439 22010 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:35:58.191485 22015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:58.176708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50043 (local address 127.0.98.62:46807)
0618 23:35:58.176751 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:58.176753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:58.183919 (+ 7166us) server_negotiation.cc:407] Connection header received
0618 23:35:58.184040 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:58.184042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:58.184058 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:58.184174 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:58.184913 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:58.185919 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:58.186702 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:58.186970 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:58.187061 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:58.188483 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:58.188485 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:58.188486 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:35:58.188494 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:58.190237 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:58.190640 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:58.190641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:58.190642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:58.190706 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:35:58.190929 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:35:58.190930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:35:58.190930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:35:58.191053 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:58.191186 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:58.191298 (+ 112us) server_negotiation.cc:299] Negotiation successful
0618 23:35:58.191366 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260618 23:35:58.193917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 21935
W20260618 23:35:58.199005 21920 connection.cc:535] client connection to 127.0.98.62:46807 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:46807): Connection reset by peer (error 104)
2026-06-18T23:35:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1346 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:35:58 dist-test-slave-6224 krb5kdc[22034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:35:58 dist-test-slave-6224 krb5kdc[22034](info): set up 2 sockets
Jun 18 23:35:58 dist-test-slave-6224 krb5kdc[22034](info): commencing operation
krb5kdc: starting...
W20260618 23:35:59.239831 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:35:59 dist-test-slave-6224 krb5kdc[22034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:35:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:35:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:35:59.366199 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46791
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:35:59.449311 22047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:35:59.449554 22047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:35:59.449579 22047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:35:59.450747 22047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:35:59.450798 22047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:35:59.450812 22047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:35:59.450825 22047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:35:59.450838 22047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:35:59.452410 22047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46791
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:35:59.452678 22047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:35:59.452934 22047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:35:59.458058 22055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:59.458067 22053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:35:59.458139 22052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:35:59.458632 22047 server_base.cc:1034] running on GCE node
I20260618 23:35:59.458909 22047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:35:59.459211 22047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:35:59.460378 22047 hybrid_clock.cc:648] HybridClock initialized: now 1781825759460319 us; error 83 us; skew 500 ppm
Jun 18 23:35:59 dist-test-slave-6224 krb5kdc[22034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825759, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:35:59.463042 22047 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:35:59.463627 22047 webserver.cc:460] Webserver started at http://127.0.98.62:35557/ using document root <none> and password file <none>
I20260618 23:35:59.463861 22047 fs_manager.cc:359] Metadata directory not provided
I20260618 23:35:59.463928 22047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:35:59.464071 22047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:35:59.465157 22047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "8819ded5828c47c3a69459fb283eb529"
format_stamp: "Formatted at 2026-06-18 23:35:59 on dist-test-slave-6224"
server_key: "d791bfba18bf496d74d8e80113aa77d2"
server_key_iv: "a4568eb563fa382c7128d172c6a3ce5a"
server_key_version: "encryptionkey@0"
I20260618 23:35:59.465533 22047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "8819ded5828c47c3a69459fb283eb529"
format_stamp: "Formatted at 2026-06-18 23:35:59 on dist-test-slave-6224"
server_key: "d791bfba18bf496d74d8e80113aa77d2"
server_key_iv: "a4568eb563fa382c7128d172c6a3ce5a"
server_key_version: "encryptionkey@0"
I20260618 23:35:59.467024 22047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:35:59.468003 22062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:35:59.468256 22047 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:35:59.468367 22047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "8819ded5828c47c3a69459fb283eb529"
format_stamp: "Formatted at 2026-06-18 23:35:59 on dist-test-slave-6224"
server_key: "d791bfba18bf496d74d8e80113aa77d2"
server_key_iv: "a4568eb563fa382c7128d172c6a3ce5a"
server_key_version: "encryptionkey@0"
I20260618 23:35:59.468466 22047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:35:59.481333 22047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:35:59.484095 22047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:35:59.484294 22047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:35:59.489013 22047 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46791
I20260618 23:35:59.489089 22114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46791 every 8 connection(s)
I20260618 23:35:59.489490 22047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:35:59.490485 22115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:35:59.492053 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 22047
I20260618 23:35:59.492195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:35:59.492398 392 external_mini_cluster.cc:1428] Setting key fdbb9590329563475ef2c22b39805df8
I20260618 23:35:59.493448 22115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529: Bootstrap starting.
I20260618 23:35:59.494088 22115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529: Neither blocks nor log segments found. Creating new log.
I20260618 23:35:59.494433 22115 log.cc:824] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529: Log is configured to *not* fsync() on all Append() calls
I20260618 23:35:59.495196 22115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529: No bootstrap required, opened a new log
I20260618 23:35:59.497668 22115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8819ded5828c47c3a69459fb283eb529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46791 } }
I20260618 23:35:59.497859 22115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:35:59.497924 22115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8819ded5828c47c3a69459fb283eb529, State: Initialized, Role: FOLLOWER
I20260618 23:35:59.498099 22115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8819ded5828c47c3a69459fb283eb529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46791 } }
I20260618 23:35:59.498199 22115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:35:59.498238 22115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:35:59.498302 22115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:35:59.499019 22115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8819ded5828c47c3a69459fb283eb529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46791 } }
I20260618 23:35:59.499131 22115 leader_election.cc:302] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8819ded5828c47c3a69459fb283eb529; no voters:
I20260618 23:35:59.499184 22118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:59.493069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47949 (local address 127.0.98.62:46791)
0618 23:35:59.493258 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:59.493268 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:59.493313 (+ 45us) server_negotiation.cc:407] Connection header received
0618 23:35:59.493972 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:59.493982 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:59.494040 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:59.494239 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:59.494576 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:59.495699 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:59.496836 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:59.497271 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:59.497367 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:59.498295 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:59.498300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:59.498301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:59.498555 (+ 254us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:59.498592 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:59.498605 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:59.498678 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:59.498848 (+ 170us) server_negotiation.cc:299] Negotiation successful
0618 23:35:59.498884 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":75,"threads_started":1}
I20260618 23:35:59.499629 22115 leader_election.cc:288] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:35:59.499749 22120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:35:59.499933 22115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:35:59.499993 22120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529 [term 1 LEADER]: Becoming Leader. State: Replica: 8819ded5828c47c3a69459fb283eb529, State: Running, Role: LEADER
I20260618 23:35:59.500275 22120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8819ded5828c47c3a69459fb283eb529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46791 } }
I20260618 23:35:59.500326 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:35:59.500741 22121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8819ded5828c47c3a69459fb283eb529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8819ded5828c47c3a69459fb283eb529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46791 } } }
I20260618 23:35:59.500828 22121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529 [sys.catalog]: This master's current role is: LEADER
I20260618 23:35:59.501331 22122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8819ded5828c47c3a69459fb283eb529. Latest consensus state: current_term: 1 leader_uuid: "8819ded5828c47c3a69459fb283eb529" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8819ded5828c47c3a69459fb283eb529" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46791 } } }
I20260618 23:35:59.501415 22122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529 [sys.catalog]: This master's current role is: LEADER
W20260618 23:35:59.502357 22135 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:35:59.502421 22135 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:35:59.502475 22129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:35:59.502640 22129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:35:59.505602 22118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:35:59.502223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44077 (local address 127.0.98.62:46791)
0618 23:35:59.502265 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0618 23:35:59.502268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:35:59.502461 (+ 193us) server_negotiation.cc:407] Connection header received
0618 23:35:59.502567 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:35:59.502568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:35:59.502580 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:35:59.502632 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:35:59.502840 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:59.503416 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:59.504272 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:35:59.504506 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:35:59.504590 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:35:59.505329 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:35:59.505330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:35:59.505331 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:35:59.505335 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:35:59.505349 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:35:59.505353 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:35:59.505407 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0618 23:35:59.505490 (+ 83us) server_negotiation.cc:299] Negotiation successful
0618 23:35:59.505500 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260618 23:35:59.508014 22129 catalog_manager.cc:1347] Generated new cluster ID: 7b1630ce6580420f9e589e30d1b4af46
I20260618 23:35:59.508085 22129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:35:59.516465 22129 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:35:59.517117 22129 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:35:59.523910 22129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8819ded5828c47c3a69459fb283eb529: Generated new TSK 0
I20260618 23:35:59.524134 22129 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:35:59.527336 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 22047
W20260618 23:35:59.532701 22032 connection.cc:535] client connection to 127.0.98.62:46791 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:46791): Connection reset by peer (error 104)
2026-06-18T23:35:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:35:59 dist-test-slave-6224 krb5kdc[22146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:35:59 dist-test-slave-6224 krb5kdc[22146](info): set up 2 sockets
Jun 18 23:35:59 dist-test-slave-6224 krb5kdc[22146](info): commencing operation
krb5kdc: starting...
W20260618 23:36:00.579841 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:36:00 dist-test-slave-6224 krb5kdc[22146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825760, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:36:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:36:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:36:00.712814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46483
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:36:00.796399 22159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:36:00.796657 22159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:36:00.796694 22159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:36:00.797856 22159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:36:00.797935 22159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:36:00.797961 22159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:36:00.797989 22159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:36:00.798020 22159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:36:00.799513 22159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46483
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:36:00.799832 22159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:36:00.800073 22159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:36:00.805712 22159 server_base.cc:1034] running on GCE node
W20260618 23:36:00.805732 22164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:36:00.805930 22167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:36:00.805724 22165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:36:00.806411 22159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:36:00.806694 22159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:36:00.807863 22159 hybrid_clock.cc:648] HybridClock initialized: now 1781825760807836 us; error 49 us; skew 500 ppm
Jun 18 23:36:00 dist-test-slave-6224 krb5kdc[22146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825760, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:36:00.810539 22159 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:36:00.811060 22159 webserver.cc:460] Webserver started at http://127.0.98.62:41805/ using document root <none> and password file <none>
I20260618 23:36:00.811319 22159 fs_manager.cc:359] Metadata directory not provided
I20260618 23:36:00.811390 22159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:36:00.811595 22159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:36:00.812703 22159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "39c9c60204284a6dafa6cd914e58f3df"
format_stamp: "Formatted at 2026-06-18 23:36:00 on dist-test-slave-6224"
server_key: "f24ffa3680c461158f0d8630d1dd81b6"
server_key_iv: "68c06fc7027ee95542ab2795f0f62bd6"
server_key_version: "encryptionkey@0"
I20260618 23:36:00.813064 22159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "39c9c60204284a6dafa6cd914e58f3df"
format_stamp: "Formatted at 2026-06-18 23:36:00 on dist-test-slave-6224"
server_key: "f24ffa3680c461158f0d8630d1dd81b6"
server_key_iv: "68c06fc7027ee95542ab2795f0f62bd6"
server_key_version: "encryptionkey@0"
I20260618 23:36:00.814637 22159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:36:00.815506 22174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:36:00.815827 22159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:36:00.815937 22159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "39c9c60204284a6dafa6cd914e58f3df"
format_stamp: "Formatted at 2026-06-18 23:36:00 on dist-test-slave-6224"
server_key: "f24ffa3680c461158f0d8630d1dd81b6"
server_key_iv: "68c06fc7027ee95542ab2795f0f62bd6"
server_key_version: "encryptionkey@0"
I20260618 23:36:00.816033 22159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:36:00.819355 22159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:36:00.821487 22159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:36:00.821648 22159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:36:00.826227 22159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46483
I20260618 23:36:00.826371 22226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46483 every 8 connection(s)
I20260618 23:36:00.826663 22159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:36:00.827762 22227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:36:00.828881 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 22159
I20260618 23:36:00.828984 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:36:00.829200 392 external_mini_cluster.cc:1428] Setting key d865d01caaee4b3fa527ac1afbf7ab9c
I20260618 23:36:00.830452 22227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df: Bootstrap starting.
I20260618 23:36:00.831173 22227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df: Neither blocks nor log segments found. Creating new log.
I20260618 23:36:00.831486 22227 log.cc:824] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df: Log is configured to *not* fsync() on all Append() calls
I20260618 23:36:00.832271 22230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:00.829866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53145 (local address 127.0.98.62:46483)
0618 23:36:00.830142 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:00.830152 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:00.830188 (+ 36us) server_negotiation.cc:407] Connection header received
0618 23:36:00.830733 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:00.830743 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:00.830780 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:00.830956 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:00.831757 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:00.831761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:00.831761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:36:00.831766 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:00.831806 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:36:00.831826 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:00.831904 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:00.832010 (+ 106us) server_negotiation.cc:299] Negotiation successful
0618 23:36:00.832066 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":134,"threads_started":1}
I20260618 23:36:00.832454 22227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df: No bootstrap required, opened a new log
I20260618 23:36:00.833354 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:36:00.835413 22227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c9c60204284a6dafa6cd914e58f3df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } }
I20260618 23:36:00.835712 22227 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:36:00.835783 22227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c9c60204284a6dafa6cd914e58f3df, State: Initialized, Role: FOLLOWER
I20260618 23:36:00.835951 22227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c9c60204284a6dafa6cd914e58f3df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } }
I20260618 23:36:00.836066 22227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:36:00.836094 22227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:36:00.836174 22227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:36:00.836683 22230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:00.835357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53969 (local address 127.0.98.62:46483)
0618 23:36:00.835400 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:00.835402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:00.835591 (+ 189us) server_negotiation.cc:407] Connection header received
0618 23:36:00.835627 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:00.835627 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:00.835632 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:00.835723 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:00.836423 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:00.836424 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:00.836424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:36:00.836428 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:00.836441 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:36:00.836445 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:00.836490 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:00.836577 (+ 87us) server_negotiation.cc:299] Negotiation successful
0618 23:36:00.836588 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260618 23:36:00.837134 22227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c9c60204284a6dafa6cd914e58f3df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } }
I20260618 23:36:00.837266 22227 leader_election.cc:302] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39c9c60204284a6dafa6cd914e58f3df; no voters:
I20260618 23:36:00.837443 22227 leader_election.cc:288] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:36:00.837522 22237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:36:00.837723 22237 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df [term 1 LEADER]: Becoming Leader. State: Replica: 39c9c60204284a6dafa6cd914e58f3df, State: Running, Role: LEADER
I20260618 23:36:00.837718 22227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:36:00.837952 22237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c9c60204284a6dafa6cd914e58f3df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } }
I20260618 23:36:00.838438 22238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39c9c60204284a6dafa6cd914e58f3df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c9c60204284a6dafa6cd914e58f3df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } } }
I20260618 23:36:00.838455 22239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39c9c60204284a6dafa6cd914e58f3df. Latest consensus state: current_term: 1 leader_uuid: "39c9c60204284a6dafa6cd914e58f3df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c9c60204284a6dafa6cd914e58f3df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46483 } } }
I20260618 23:36:00.838716 22238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df [sys.catalog]: This master's current role is: LEADER
I20260618 23:36:00.838727 22239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df [sys.catalog]: This master's current role is: LEADER
I20260618 23:36:00.839002 22246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:36:00.839540 22246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:36:00.841428 22246 catalog_manager.cc:1347] Generated new cluster ID: 3e4e07e2b42b43018282bd83e416e8e4
I20260618 23:36:00.841490 22246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:36:00.861168 22246 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:36:00.861424 22246 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:36:00.869827 22246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39c9c60204284a6dafa6cd914e58f3df: Generated new TSK 0
I20260618 23:36:00.870065 22246 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:36:00.876406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 22159
2026-06-18T23:36:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:36:00 dist-test-slave-6224 krb5kdc[22258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:36:00 dist-test-slave-6224 krb5kdc[22258](info): set up 2 sockets
Jun 18 23:36:00 dist-test-slave-6224 krb5kdc[22258](info): commencing operation
krb5kdc: starting...
W20260618 23:36:02.003806 392 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:36:02 dist-test-slave-6224 krb5kdc[22258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:36:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:36:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:36:02.135037 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34257
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:36:02.217207 22271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:36:02.217571 22271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:36:02.217614 22271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:36:02.218794 22271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:36:02.218875 22271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:36:02.218904 22271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:36:02.218935 22271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:36:02.218966 22271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:36:02.220494 22271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34257
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:36:02.220778 22271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:36:02.221011 22271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:36:02.226439 22277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:36:02.226632 22279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:36:02.226465 22276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:36:02.226769 22271 server_base.cc:1034] running on GCE node
I20260618 23:36:02.227174 22271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:36:02.227789 22271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:36:02.228997 22271 hybrid_clock.cc:648] HybridClock initialized: now 1781825762228970 us; error 43 us; skew 500 ppm
Jun 18 23:36:02 dist-test-slave-6224 krb5kdc[22258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:36:02.232216 22271 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:36:02.232754 22271 webserver.cc:460] Webserver started at http://127.0.98.62:37057/ using document root <none> and password file <none>
I20260618 23:36:02.233029 22271 fs_manager.cc:359] Metadata directory not provided
I20260618 23:36:02.233103 22271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:36:02.233270 22271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:36:02.234483 22271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "a7132aff49c84d65b7a2a6c71c7bb618"
format_stamp: "Formatted at 2026-06-18 23:36:02 on dist-test-slave-6224"
server_key: "72585bdf0416cacfd819eda8d884fc3b"
server_key_iv: "0fd67e8c5650882981e87c0fa857081f"
server_key_version: "encryptionkey@0"
I20260618 23:36:02.234866 22271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "a7132aff49c84d65b7a2a6c71c7bb618"
format_stamp: "Formatted at 2026-06-18 23:36:02 on dist-test-slave-6224"
server_key: "72585bdf0416cacfd819eda8d884fc3b"
server_key_iv: "0fd67e8c5650882981e87c0fa857081f"
server_key_version: "encryptionkey@0"
I20260618 23:36:02.236424 22271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:36:02.237265 22286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:36:02.237504 22271 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:36:02.237618 22271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "a7132aff49c84d65b7a2a6c71c7bb618"
format_stamp: "Formatted at 2026-06-18 23:36:02 on dist-test-slave-6224"
server_key: "72585bdf0416cacfd819eda8d884fc3b"
server_key_iv: "0fd67e8c5650882981e87c0fa857081f"
server_key_version: "encryptionkey@0"
I20260618 23:36:02.237702 22271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:36:02.252388 22271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:36:02.254523 22271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:36:02.254712 22271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:36:02.260248 22271 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34257
I20260618 23:36:02.260339 22338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34257 every 8 connection(s)
I20260618 23:36:02.260784 22271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:36:02.261268 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 22271
I20260618 23:36:02.261410 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:36:02.261720 392 external_mini_cluster.cc:1428] Setting key 587271f52e3ce0e5f233c782f2aed611
I20260618 23:36:02.261912 22339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:36:02.265563 22339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618: Bootstrap starting.
I20260618 23:36:02.266270 22339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618: Neither blocks nor log segments found. Creating new log.
I20260618 23:36:02.266587 22339 log.cc:824] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618: Log is configured to *not* fsync() on all Append() calls
I20260618 23:36:02.267524 22339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618: No bootstrap required, opened a new log
Jun 18 23:36:02 dist-test-slave-6224 krb5kdc[22258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:36:02.270745 22339 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7132aff49c84d65b7a2a6c71c7bb618" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34257 } }
I20260618 23:36:02.270967 22339 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:36:02.271020 22339 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7132aff49c84d65b7a2a6c71c7bb618, State: Initialized, Role: FOLLOWER
I20260618 23:36:02.271229 22339 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7132aff49c84d65b7a2a6c71c7bb618" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34257 } }
I20260618 23:36:02.271338 22339 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:36:02.271374 22339 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:36:02.271427 22339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:36:02.272364 22339 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7132aff49c84d65b7a2a6c71c7bb618" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34257 } }
I20260618 23:36:02.272511 22339 leader_election.cc:302] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7132aff49c84d65b7a2a6c71c7bb618; no voters:
I20260618 23:36:02.272766 22339 leader_election.cc:288] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:36:02.272866 22344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:36:02.273073 22344 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618 [term 1 LEADER]: Becoming Leader. State: Replica: a7132aff49c84d65b7a2a6c71c7bb618, State: Running, Role: LEADER
I20260618 23:36:02.273175 22339 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:36:02.273284 22344 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7132aff49c84d65b7a2a6c71c7bb618" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34257 } }
I20260618 23:36:02.273713 22344 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7132aff49c84d65b7a2a6c71c7bb618. Latest consensus state: current_term: 1 leader_uuid: "a7132aff49c84d65b7a2a6c71c7bb618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7132aff49c84d65b7a2a6c71c7bb618" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34257 } } }
I20260618 23:36:02.273783 22344 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618 [sys.catalog]: This master's current role is: LEADER
I20260618 23:36:02.274454 22349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:36:02.275089 22349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:36:02.276136 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:02.262390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46145 (local address 127.0.98.62:34257)
0618 23:36:02.262646 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:02.262656 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:02.262696 (+ 40us) server_negotiation.cc:407] Connection header received
0618 23:36:02.263323 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:02.263332 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:02.263405 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:02.263624 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:02.264573 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:02.266005 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:02.267053 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:02.267496 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:02.267701 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:02.270834 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:02.270838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:02.270838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:02.270853 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:02.273171 (+ 2318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:02.274000 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:02.274005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:02.274006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:02.274171 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:02.274478 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:02.274479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:02.274479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:02.274916 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:02.275719 (+ 803us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:02.275789 (+ 70us) server_negotiation.cc:299] Negotiation successful
0618 23:36:02.275919 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":125,"threads_started":1}
I20260618 23:36:02.276773 22345 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7132aff49c84d65b7a2a6c71c7bb618" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7132aff49c84d65b7a2a6c71c7bb618" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34257 } } }
I20260618 23:36:02.277104 22345 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618 [sys.catalog]: This master's current role is: LEADER
I20260618 23:36:02.277204 22349 catalog_manager.cc:1347] Generated new cluster ID: e3c1506903d14898840a5a6dc792386d
I20260618 23:36:02.277277 22349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:36:02.277369 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:36:02.290701 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:02.279035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53461 (local address 127.0.98.62:34257)
0618 23:36:02.279077 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:02.279079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:02.283833 (+ 4754us) server_negotiation.cc:407] Connection header received
0618 23:36:02.283915 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:02.283917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:02.283929 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:02.284022 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:02.284707 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:02.285309 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:02.286103 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:02.286342 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:02.286431 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:02.287953 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:02.287954 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:02.287955 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:02.287963 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:02.289491 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:02.289861 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:02.289862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:02.289863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:02.289924 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:02.290120 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:02.290121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:02.290121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:02.290257 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:02.290373 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:02.290477 (+ 104us) server_negotiation.cc:299] Negotiation successful
0618 23:36:02.290569 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260618 23:36:02.304304 22349 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:36:02.304991 22349 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:36:02.310528 22349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7132aff49c84d65b7a2a6c71c7bb618: Generated new TSK 0
I20260618 23:36:02.310791 22349 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:36:02.315922 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 22271
2026-06-18T23:36:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:36:02 dist-test-slave-6224 krb5kdc[22370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:36:02 dist-test-slave-6224 krb5kdc[22370](info): set up 2 sockets
Jun 18 23:36:02 dist-test-slave-6224 krb5kdc[22370](info): commencing operation
krb5kdc: starting...
W20260618 23:36:03.384197 392 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:36:03 dist-test-slave-6224 krb5kdc[22370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:36:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:36:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:36:03.519188 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40361
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:36:03.605042 22383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:36:03.605319 22383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:36:03.605362 22383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:36:03.606675 22383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:36:03.606770 22383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:36:03.606798 22383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:36:03.606822 22383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:36:03.606855 22383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:36:03.608546 22383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:36:03.608856 22383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:36:03.609125 22383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:36:03.614887 22389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:36:03.614904 22388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:36:03.614936 22391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:36:03.615391 22383 server_base.cc:1034] running on GCE node
I20260618 23:36:03.615810 22383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:36:03.616218 22383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:36:03.617422 22383 hybrid_clock.cc:648] HybridClock initialized: now 1781825763617395 us; error 59 us; skew 500 ppm
Jun 18 23:36:03 dist-test-slave-6224 krb5kdc[22370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825763, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:36:03.626164 22383 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:36:03.626775 22383 webserver.cc:460] Webserver started at http://127.0.98.62:43179/ using document root <none> and password file <none>
I20260618 23:36:03.627076 22383 fs_manager.cc:359] Metadata directory not provided
I20260618 23:36:03.627152 22383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:36:03.627331 22383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:36:03.628667 22383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "22953d75a8324700967ba89967545758"
format_stamp: "Formatted at 2026-06-18 23:36:03 on dist-test-slave-6224"
server_key: "fdc194061c374d4011bfb3df773ee320"
server_key_iv: "043cef32c99f7f5dc4a14b3e8c3fe8ac"
server_key_version: "encryptionkey@0"
I20260618 23:36:03.629099 22383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "22953d75a8324700967ba89967545758"
format_stamp: "Formatted at 2026-06-18 23:36:03 on dist-test-slave-6224"
server_key: "fdc194061c374d4011bfb3df773ee320"
server_key_iv: "043cef32c99f7f5dc4a14b3e8c3fe8ac"
server_key_version: "encryptionkey@0"
I20260618 23:36:03.630759 22383 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:36:03.631834 22398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:36:03.632089 22383 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:36:03.632215 22383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "22953d75a8324700967ba89967545758"
format_stamp: "Formatted at 2026-06-18 23:36:03 on dist-test-slave-6224"
server_key: "fdc194061c374d4011bfb3df773ee320"
server_key_iv: "043cef32c99f7f5dc4a14b3e8c3fe8ac"
server_key_version: "encryptionkey@0"
I20260618 23:36:03.632318 22383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:36:03.664279 22383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:36:03.666811 22383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:36:03.666934 22383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:36:03.671414 22383 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40361
I20260618 23:36:03.671515 22450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40361 every 8 connection(s)
I20260618 23:36:03.671906 22383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:36:03.672987 22451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:36:03.676011 22451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758: Bootstrap starting.
I20260618 23:36:03.676205 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 22383
I20260618 23:36:03.676375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:36:03.676630 392 external_mini_cluster.cc:1428] Setting key d7ebbe2c361d676a3b9599f55d14c90a
I20260618 23:36:03.676928 22451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758: Neither blocks nor log segments found. Creating new log.
I20260618 23:36:03.677331 22451 log.cc:824] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758: Log is configured to *not* fsync() on all Append() calls
I20260618 23:36:03.678380 22451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758: No bootstrap required, opened a new log
I20260618 23:36:03.681173 22451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22953d75a8324700967ba89967545758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40361 } }
I20260618 23:36:03.681347 22451 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:36:03.681373 22451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22953d75a8324700967ba89967545758, State: Initialized, Role: FOLLOWER
I20260618 23:36:03.681538 22451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22953d75a8324700967ba89967545758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40361 } }
I20260618 23:36:03.681651 22451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:36:03.681703 22451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:36:03.681766 22451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:36:03.682498 22451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22953d75a8324700967ba89967545758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40361 } }
I20260618 23:36:03.682650 22451 leader_election.cc:302] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22953d75a8324700967ba89967545758; no voters:
I20260618 23:36:03.682904 22451 leader_election.cc:288] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:36:03.682941 22456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:36:03.683166 22456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758 [term 1 LEADER]: Becoming Leader. State: Replica: 22953d75a8324700967ba89967545758, State: Running, Role: LEADER
I20260618 23:36:03.683342 22451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:36:03.683332 22456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22953d75a8324700967ba89967545758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40361 } }
I20260618 23:36:03.683881 22456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22953d75a8324700967ba89967545758. Latest consensus state: current_term: 1 leader_uuid: "22953d75a8324700967ba89967545758" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22953d75a8324700967ba89967545758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40361 } } }
I20260618 23:36:03.683981 22454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:03.677389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52697 (local address 127.0.98.62:40361)
0618 23:36:03.677662 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:03.677673 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:03.677715 (+ 42us) server_negotiation.cc:407] Connection header received
0618 23:36:03.678435 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:03.678446 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:03.678525 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:03.678725 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:03.678989 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:03.680537 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:03.681609 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:03.682232 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:03.682379 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:03.683155 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:03.683160 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:03.683160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:36:03.683392 (+ 232us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:03.683432 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:36:03.683449 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:03.683505 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:03.683678 (+ 173us) server_negotiation.cc:299] Negotiation successful
0618 23:36:03.683762 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":108,"threads_started":1}
I20260618 23:36:03.683892 22458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22953d75a8324700967ba89967545758" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22953d75a8324700967ba89967545758" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40361 } } }
I20260618 23:36:03.684093 22456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758 [sys.catalog]: This master's current role is: LEADER
I20260618 23:36:03.684494 22458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758 [sys.catalog]: This master's current role is: LEADER
I20260618 23:36:03.684643 22465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:36:03.685150 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:36:03.685463 22465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:36:03.687975 22465 catalog_manager.cc:1347] Generated new cluster ID: 21ef8088250a4e53a8e3c5bf7072bc5e
I20260618 23:36:03.688035 22465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:36:03.690927 22454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:03.686977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46461 (local address 127.0.98.62:40361)
0618 23:36:03.687014 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:03.687016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:03.687156 (+ 140us) server_negotiation.cc:407] Connection header received
0618 23:36:03.687252 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:03.687253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:03.687263 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:03.687296 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:03.687541 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:03.688319 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:03.689378 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:03.689694 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:03.689793 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:03.690611 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:03.690613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:03.690613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:36:03.690620 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:03.690643 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:36:03.690649 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:03.690729 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:03.690802 (+ 73us) server_negotiation.cc:299] Negotiation successful
0618 23:36:03.690835 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260618 23:36:03.715049 22465 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:36:03.715708 22465 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:36:03.728524 22465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22953d75a8324700967ba89967545758: Generated new TSK 0
I20260618 23:36:03.728744 22465 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:36:03.737471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 22383
2026-06-18T23:36:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1421 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:36:03 dist-test-slave-6224 krb5kdc[22482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:36:03 dist-test-slave-6224 krb5kdc[22482](info): set up 2 sockets
Jun 18 23:36:03 dist-test-slave-6224 krb5kdc[22482](info): commencing operation
krb5kdc: starting...
W20260618 23:36:04.783953 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:36:04 dist-test-slave-6224 krb5kdc[22482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:36:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:36:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:36:04.910521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46465
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:36:05.001533 22495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:36:05.001853 22495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:36:05.001897 22495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:36:05.003328 22495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:36:05.003420 22495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:36:05.003448 22495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:36:05.003474 22495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:36:05.003506 22495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:36:05.005391 22495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:36:05.005776 22495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:36:05.006085 22495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:36:05.012053 22503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:36:05.012068 22501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:36:05.012210 22495 server_base.cc:1034] running on GCE node
W20260618 23:36:05.012096 22500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:36:05.012663 22495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:36:05.013021 22495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:36:05.014210 22495 hybrid_clock.cc:648] HybridClock initialized: now 1781825765014177 us; error 77 us; skew 500 ppm
Jun 18 23:36:05 dist-test-slave-6224 krb5kdc[22482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:36:05.017164 22495 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:36:05.017663 22495 webserver.cc:460] Webserver started at http://127.0.98.62:34657/ using document root <none> and password file <none>
I20260618 23:36:05.017930 22495 fs_manager.cc:359] Metadata directory not provided
I20260618 23:36:05.018008 22495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:36:05.018165 22495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:36:05.019287 22495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "beb446029e854f01b947d7e6bb6a772f"
format_stamp: "Formatted at 2026-06-18 23:36:05 on dist-test-slave-6224"
server_key: "747b17dd40eacbe751b8c3fd997c0ab9"
server_key_iv: "f2e4df74eb7b9018aded43996c08aeb8"
server_key_version: "encryptionkey@0"
I20260618 23:36:05.019721 22495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "beb446029e854f01b947d7e6bb6a772f"
format_stamp: "Formatted at 2026-06-18 23:36:05 on dist-test-slave-6224"
server_key: "747b17dd40eacbe751b8c3fd997c0ab9"
server_key_iv: "f2e4df74eb7b9018aded43996c08aeb8"
server_key_version: "encryptionkey@0"
I20260618 23:36:05.021220 22495 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:36:05.022151 22510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:36:05.022519 22495 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:36:05.022640 22495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "beb446029e854f01b947d7e6bb6a772f"
format_stamp: "Formatted at 2026-06-18 23:36:05 on dist-test-slave-6224"
server_key: "747b17dd40eacbe751b8c3fd997c0ab9"
server_key_iv: "f2e4df74eb7b9018aded43996c08aeb8"
server_key_version: "encryptionkey@0"
I20260618 23:36:05.022720 22495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:36:05.026366 22495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:36:05.028584 22495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:36:05.028743 22495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:36:05.033095 22495 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46465
I20260618 23:36:05.033154 22562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46465 every 8 connection(s)
I20260618 23:36:05.033527 22495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:36:05.034705 22563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:36:05.035974 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 22495
I20260618 23:36:05.036113 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:36:05.036301 392 external_mini_cluster.cc:1428] Setting key 5e513df76ac0e1cd7b92e9d7b3562093
I20260618 23:36:05.038079 22563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f: Bootstrap starting.
I20260618 23:36:05.038931 22563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f: Neither blocks nor log segments found. Creating new log.
I20260618 23:36:05.039305 22563 log.cc:824] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f: Log is configured to *not* fsync() on all Append() calls
I20260618 23:36:05.039422 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:05.036915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43493 (local address 127.0.98.62:46465)
0618 23:36:05.037124 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:05.037135 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:05.037237 (+ 102us) server_negotiation.cc:407] Connection header received
0618 23:36:05.037819 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:05.037829 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:05.037868 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:05.038051 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:05.038843 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:05.038847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:05.038847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:36:05.038852 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:05.038892 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:36:05.038908 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:05.039017 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:05.039136 (+ 119us) server_negotiation.cc:299] Negotiation successful
0618 23:36:05.039194 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":73,"threads_started":1}
I20260618 23:36:05.040069 22563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f: No bootstrap required, opened a new log
I20260618 23:36:05.040541 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:36:05.042495 22563 raft_consensus.cc:348] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beb446029e854f01b947d7e6bb6a772f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46465 } }
I20260618 23:36:05.042699 22563 raft_consensus.cc:374] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:36:05.042749 22563 raft_consensus.cc:729] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beb446029e854f01b947d7e6bb6a772f, State: Initialized, Role: FOLLOWER
I20260618 23:36:05.042882 22563 consensus_queue.cc:260] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beb446029e854f01b947d7e6bb6a772f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46465 } }
I20260618 23:36:05.042989 22563 raft_consensus.cc:388] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:36:05.043028 22563 raft_consensus.cc:482] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:36:05.043095 22563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:36:05.043471 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:05.042043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36015 (local address 127.0.98.62:46465)
0618 23:36:05.042102 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:05.042104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:05.042278 (+ 174us) server_negotiation.cc:407] Connection header received
0618 23:36:05.042350 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:05.042350 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:05.042356 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:05.042414 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:05.043242 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:05.043243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:05.043243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:36:05.043248 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:05.043262 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:36:05.043266 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:05.043321 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:05.043403 (+ 82us) server_negotiation.cc:299] Negotiation successful
0618 23:36:05.043416 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260618 23:36:05.043936 22563 raft_consensus.cc:504] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beb446029e854f01b947d7e6bb6a772f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46465 } }
I20260618 23:36:05.044076 22563 leader_election.cc:302] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: beb446029e854f01b947d7e6bb6a772f; no voters:
I20260618 23:36:05.044250 22563 leader_election.cc:288] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:36:05.044333 22573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:36:05.044548 22573 raft_consensus.cc:686] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f [term 1 LEADER]: Becoming Leader. State: Replica: beb446029e854f01b947d7e6bb6a772f, State: Running, Role: LEADER
I20260618 23:36:05.044554 22563 sys_catalog.cc:564] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:36:05.044679 22573 consensus_queue.cc:237] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beb446029e854f01b947d7e6bb6a772f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46465 } }
I20260618 23:36:05.045130 22575 sys_catalog.cc:455] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f [sys.catalog]: SysCatalogTable state changed. Reason: New leader beb446029e854f01b947d7e6bb6a772f. Latest consensus state: current_term: 1 leader_uuid: "beb446029e854f01b947d7e6bb6a772f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beb446029e854f01b947d7e6bb6a772f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46465 } } }
I20260618 23:36:05.045161 22574 sys_catalog.cc:455] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "beb446029e854f01b947d7e6bb6a772f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beb446029e854f01b947d7e6bb6a772f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46465 } } }
I20260618 23:36:05.045317 22575 sys_catalog.cc:458] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f [sys.catalog]: This master's current role is: LEADER
I20260618 23:36:05.045341 22574 sys_catalog.cc:458] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f [sys.catalog]: This master's current role is: LEADER
I20260618 23:36:05.045680 22582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:36:05.046278 22582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:36:05.048440 22582 catalog_manager.cc:1347] Generated new cluster ID: 4516c6f7c3744328a4c6435dbe7c4a61
I20260618 23:36:05.048569 22582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:36:05.054773 22582 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:36:05.054989 22582 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:36:05.064860 22582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P beb446029e854f01b947d7e6bb6a772f: Generated new TSK 0
I20260618 23:36:05.065099 22582 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:36:05.067494 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 22495
2026-06-18T23:36:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1330 ms)
[----------] 18 tests from AuthTokenIssuingTest (24461 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:36:05 dist-test-slave-6224 krb5kdc[22594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:36:05 dist-test-slave-6224 krb5kdc[22594](info): set up 2 sockets
Jun 18 23:36:05 dist-test-slave-6224 krb5kdc[22594](info): commencing operation
krb5kdc: starting...
W20260618 23:36:06.111788 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:36:06 dist-test-slave-6224 krb5kdc[22594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:36:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:36:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:36:06.242888 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44475
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44475,127.0.98.61:40729,127.0.98.60:35347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:36:06.329010 22607 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 23:36:06.329293 22607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:36:06.329325 22607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:36:06.329368 22607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:36:06.330695 22607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:36:06.330782 22607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:36:06.330809 22607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:36:06.330829 22607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:36:06.330842 22607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:36:06.332376 22607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44475,127.0.98.61:40729,127.0.98.60:35347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:36:06.332583 22607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:36:06.332763 22607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:36:06.338131 22613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:36:06.338172 22612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:36:06.338294 22607 server_base.cc:1034] running on GCE node
W20260618 23:36:06.338129 22615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:36:06.338827 22607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:36:06.339362 22607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:36:06.340557 22607 hybrid_clock.cc:648] HybridClock initialized: now 1781825766340533 us; error 44 us; skew 500 ppm
Jun 18 23:36:06 dist-test-slave-6224 krb5kdc[22594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:36:06.343667 22607 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:36:06.344203 22607 webserver.cc:460] Webserver started at http://127.0.98.62:42277/ using document root <none> and password file <none>
I20260618 23:36:06.344480 22607 fs_manager.cc:359] Metadata directory not provided
I20260618 23:36:06.344553 22607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:36:06.344713 22607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:36:06.345932 22607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "b7786b29b6184665ae5471f42dbb3294"
format_stamp: "Formatted at 2026-06-18 23:36:06 on dist-test-slave-6224"
server_key: "90fba13f2e30c3da39e8ad24bb690a6b"
server_key_iv: "3d663ec7808754bff2c9a9132b9df948"
server_key_version: "encryptionkey@0"
I20260618 23:36:06.346329 22607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "b7786b29b6184665ae5471f42dbb3294"
format_stamp: "Formatted at 2026-06-18 23:36:06 on dist-test-slave-6224"
server_key: "90fba13f2e30c3da39e8ad24bb690a6b"
server_key_iv: "3d663ec7808754bff2c9a9132b9df948"
server_key_version: "encryptionkey@0"
I20260618 23:36:06.347852 22607 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:36:06.348966 22622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:36:06.349303 22607 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:36:06.349455 22607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "b7786b29b6184665ae5471f42dbb3294"
format_stamp: "Formatted at 2026-06-18 23:36:06 on dist-test-slave-6224"
server_key: "90fba13f2e30c3da39e8ad24bb690a6b"
server_key_iv: "3d663ec7808754bff2c9a9132b9df948"
server_key_version: "encryptionkey@0"
I20260618 23:36:06.349546 22607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:36:06.365407 22607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:36:06.367898 22607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:36:06.368084 22607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:36:06.373085 22607 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44475
I20260618 23:36:06.373140 22674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44475 every 8 connection(s)
I20260618 23:36:06.373615 22607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:36:06.374702 22675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:36:06.376492 22675 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:36:06.379213 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 22607
I20260618 23:36:06.379335 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:36:06.379580 392 external_mini_cluster.cc:1428] Setting key bad18b15041ae9f013c2870e91432041
Jun 18 23:36:06 dist-test-slave-6224 krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260618 23:36:06.389294 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.377282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32908 (local address 127.0.98.62:44475)
0618 23:36:06.377527 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.377541 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.377702 (+ 161us) server_negotiation.cc:407] Connection header received
0618 23:36:06.378230 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.378233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.378255 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.378370 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.379001 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.380176 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.381478 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.382048 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.382194 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.385101 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:06.385108 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:06.385108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:06.385116 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:06.386947 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.387650 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.387652 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.387653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.387736 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.388102 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.388103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.388104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.388553 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:06.388637 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:06.388935 (+ 298us) server_negotiation.cc:299] Negotiation successful
0618 23:36:06.389114 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":201,"thread_start_us":133,"threads_started":1}
I20260618 23:36:06.389315 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.377231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44475 (local address 127.0.0.1:32908)
0618 23:36:06.377454 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.377506 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0618 23:36:06.378044 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:36:06.378417 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:36:06.378421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:36:06.378684 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:36:06.378862 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:06.378866 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.380309 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.380310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:06.381359 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:06.381362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.382197 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.382198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:06.382273 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.382896 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:36:06.382904 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:36:06.384989 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:36:06.387076 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:06.387086 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:06.387090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:06.387477 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:06.387859 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:06.387860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:06.387860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:06.387988 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:06.388646 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:36:06.388655 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:36:06.388753 (+ 98us) client_negotiation.cc:769] Sending connection context
0618 23:36:06.388863 (+ 110us) client_negotiation.cc:240] Negotiation successful
0618 23:36:06.389034 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":88,"threads_started":1}
I20260618 23:36:06.390352 22675 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:36:06.390765 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.390658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40729 (local address 127.0.0.1:34936)
0618 23:36:06.390697 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.390743 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
W20260618 23:36:06.390940 22623 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40729: connect: Connection refused (error 111)
I20260618 23:36:06.391525 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.391373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40729 (local address 127.0.0.1:34942)
0618 23:36:06.391460 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.391497 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260618 23:36:06.391716 22675 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40729: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40729: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260618 23:36:06.407490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40729
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44475,127.0.98.61:40729,127.0.98.60:35347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:36:06.441054 22675 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } attempt: 1
I20260618 23:36:06.441771 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.441620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40729 (local address 127.0.0.1:34956)
0618 23:36:06.441683 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.441731 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260618 23:36:06.442175 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.442088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40729 (local address 127.0.0.1:34972)
0618 23:36:06.442129 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.442153 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260618 23:36:06.442293 22675 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40729: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40729: connect: Connection refused (error 111)
W20260618 23:36:06.500824 22682 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 23:36:06.501153 22682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:36:06.501190 22682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:36:06.501235 22682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:36:06.502557 22682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:36:06.502640 22682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:36:06.502668 22682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:36:06.502703 22682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:36:06.502728 22682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:36:06.504516 22682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44475,127.0.98.61:40729,127.0.98.60:35347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:36:06.504848 22682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:36:06.505136 22682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:36:06.510545 22675 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } attempt: 2
W20260618 23:36:06.510792 22690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:36:06.510792 22688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:36:06.511139 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.510908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40729 (local address 127.0.0.1:34980)
0618 23:36:06.511052 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.511082 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260618 23:36:06.510804 22687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:36:06.511350 22682 server_base.cc:1034] running on GCE node
I20260618 23:36:06.511742 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.511593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40729 (local address 127.0.0.1:34996)
0618 23:36:06.511671 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.511709 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260618 23:36:06.511806 22682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20260618 23:36:06.511897 22675 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40729: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40729: connect: Connection refused (error 111)
I20260618 23:36:06.512228 22682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:36:06.513417 22682 hybrid_clock.cc:648] HybridClock initialized: now 1781825766513405 us; error 34 us; skew 500 ppm
Jun 18 23:36:06 dist-test-slave-6224 krb5kdc[22594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:36:06.516450 22682 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260618 23:36:06.517108 22682 webserver.cc:460] Webserver started at http://127.0.98.61:38863/ using document root <none> and password file <none>
I20260618 23:36:06.517409 22682 fs_manager.cc:359] Metadata directory not provided
I20260618 23:36:06.517490 22682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:36:06.517648 22682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:36:06.518841 22682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-1/data/instance:
uuid: "239a1bc0227e400d9ba3a6c799509046"
format_stamp: "Formatted at 2026-06-18 23:36:06 on dist-test-slave-6224"
server_key: "5edcaae7370db32af936974550fd3f51"
server_key_iv: "44a432ece1e711a529cd5ef9611ab100"
server_key_version: "encryptionkey@0"
I20260618 23:36:06.519186 22682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-1/wal/instance:
uuid: "239a1bc0227e400d9ba3a6c799509046"
format_stamp: "Formatted at 2026-06-18 23:36:06 on dist-test-slave-6224"
server_key: "5edcaae7370db32af936974550fd3f51"
server_key_iv: "44a432ece1e711a529cd5ef9611ab100"
server_key_version: "encryptionkey@0"
I20260618 23:36:06.521090 22682 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260618 23:36:06.521975 22697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:36:06.522218 22682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:36:06.522315 22682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-1/wal
uuid: "239a1bc0227e400d9ba3a6c799509046"
format_stamp: "Formatted at 2026-06-18 23:36:06 on dist-test-slave-6224"
server_key: "5edcaae7370db32af936974550fd3f51"
server_key_iv: "44a432ece1e711a529cd5ef9611ab100"
server_key_version: "encryptionkey@0"
I20260618 23:36:06.522383 22682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:36:06.533769 22682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:36:06.536136 22682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:36:06.536264 22682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:36:06.540750 22682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40729
I20260618 23:36:06.540843 22749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40729 every 8 connection(s)
I20260618 23:36:06.541224 22682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-1/data/info.pb
I20260618 23:36:06.542456 22750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:36:06.543233 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 22682
I20260618 23:36:06.543322 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-1/wal/instance
I20260618 23:36:06.543529 392 external_mini_cluster.cc:1428] Setting key 74f680cd1d279900d31cbd6f7ad7157b
I20260618 23:36:06.544579 22750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 18 23:36:06 dist-test-slave-6224 krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:36:06.558244 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.545603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32916 (local address 127.0.98.62:44475)
0618 23:36:06.545690 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.545692 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.545997 (+ 305us) server_negotiation.cc:407] Connection header received
0618 23:36:06.546620 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.546621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.546639 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.546724 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.547453 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.548357 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.549469 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.549802 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.549901 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.554871 (+ 4970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:06.554873 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:06.554874 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:06.554882 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:06.556715 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.557185 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.557186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.557187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.557251 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.557507 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.557508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.557508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.557649 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:06.557728 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:06.558034 (+ 306us) server_negotiation.cc:299] Negotiation successful
0618 23:36:06.558135 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 23:36:06.558470 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.545563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44475 (local address 127.0.0.1:32916)
0618 23:36:06.545794 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.545859 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0618 23:36:06.546492 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:36:06.546803 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:36:06.546807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:36:06.547120 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:36:06.547349 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:06.547353 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.548475 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.548476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:06.549362 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:06.549363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.549947 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.549948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:06.550062 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.550731 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:36:06.550740 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:36:06.554730 (+ 3990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:36:06.556818 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:06.556827 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:06.556830 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:06.557086 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:06.557318 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:06.557319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:06.557320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:06.557437 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:06.557719 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:36:06.557726 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:36:06.557896 (+ 170us) client_negotiation.cc:769] Sending connection context
0618 23:36:06.558015 (+ 119us) client_negotiation.cc:240] Negotiation successful
0618 23:36:06.558200 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":110,"threads_started":1}
I20260618 23:36:06.559316 22750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 23:36:06 dist-test-slave-6224 krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260618 23:36:06.568241 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.559821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35008 (local address 127.0.98.61:40729)
0618 23:36:06.560020 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.560026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.560051 (+ 25us) server_negotiation.cc:407] Connection header received
0618 23:36:06.560075 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.560085 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.560097 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.560177 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.560667 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.561437 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.562527 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.562936 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.563042 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.565155 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:06.565165 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:06.565166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:06.565173 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:06.566923 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.567289 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.567294 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.567294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.567355 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.567574 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.567575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.567575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.567768 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:06.567854 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:06.567975 (+ 121us) server_negotiation.cc:299] Negotiation successful
0618 23:36:06.568078 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":166,"thread_start_us":124,"threads_started":1}
I20260618 23:36:06.568243 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.559746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40729 (local address 127.0.0.1:35008)
0618 23:36:06.559833 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.559851 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 23:36:06.559937 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:36:06.560218 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:36:06.560219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:36:06.560439 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:36:06.560543 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:06.560546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.561547 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.561548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:06.562437 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:06.562439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.563041 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.563042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:06.563123 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.563657 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:36:06.563664 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:36:06.565011 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:36:06.567021 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:06.567022 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:06.567023 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:06.567192 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:06.567429 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:06.567430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:06.567430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:06.567482 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:06.567829 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:36:06.567831 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:36:06.567873 (+ 42us) client_negotiation.cc:769] Sending connection context
0618 23:36:06.567945 (+ 72us) client_negotiation.cc:240] Negotiation successful
0618 23:36:06.568146 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":93}
I20260618 23:36:06.569056 22750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:36:06.569478 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.569343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35347 (local address 127.0.0.1:60788)
0618 23:36:06.569400 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.569455 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260618 23:36:06.569634 22701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35347: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260618 23:36:06.570215 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.570077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35347 (local address 127.0.0.1:60802)
0618 23:36:06.570151 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.570189 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260618 23:36:06.570400 22750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35347: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35347: connect: Connection refused (error 111)
I20260618 23:36:06.570604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35347
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44475,127.0.98.61:40729,127.0.98.60:35347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:36:06.603056 22675 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } attempt: 3
Jun 18 23:36:06 dist-test-slave-6224 krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260618 23:36:06.612762 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.603813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40729 (local address 127.0.0.1:35010)
0618 23:36:06.603874 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.603886 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:36:06.603952 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:36:06.604202 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:36:06.604203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:36:06.604457 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:36:06.604558 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:06.604560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.605665 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.605666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:06.606415 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:06.606417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.606841 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.606842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:06.606922 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.607486 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:36:06.607492 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:36:06.609056 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:36:06.611291 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:06.611294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:06.611295 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:06.611632 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:06.611960 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:06.611961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:06.611961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:06.612042 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:06.612433 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:36:06.612434 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:36:06.612488 (+ 54us) client_negotiation.cc:769] Sending connection context
0618 23:36:06.612542 (+ 54us) client_negotiation.cc:240] Negotiation successful
0618 23:36:06.612659 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260618 23:36:06.613116 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.603704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.0.98.61:40729)
0618 23:36:06.603771 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.603773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.604011 (+ 238us) server_negotiation.cc:407] Connection header received
0618 23:36:06.604049 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.604050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.604066 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.604120 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.604644 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.605551 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.606516 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.606755 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.606834 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.609168 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:06.609169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:06.609170 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:06.609179 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:06.611162 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.611788 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.611790 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.611790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.611873 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.612151 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.612154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.612154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.612340 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:06.612408 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:06.612921 (+ 513us) server_negotiation.cc:299] Negotiation successful
0618 23:36:06.613029 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260618 23:36:06.613612 22675 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:36:06.614107 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.613962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35347 (local address 127.0.0.1:60812)
0618 23:36:06.614030 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.614078 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260618 23:36:06.614540 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.614448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35347 (local address 127.0.0.1:60824)
0618 23:36:06.614492 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.614522 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260618 23:36:06.614703 22675 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35347: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35347: connect: Connection refused (error 111)
I20260618 23:36:06.619655 22750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } attempt: 1
I20260618 23:36:06.620219 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.620063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35347 (local address 127.0.0.1:60836)
0618 23:36:06.620127 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.620172 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260618 23:36:06.620561 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.620463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35347 (local address 127.0.0.1:60844)
0618 23:36:06.620505 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.620533 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260618 23:36:06.620647 22750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35347: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35347: connect: Connection refused (error 111)
I20260618 23:36:06.645920 22675 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } attempt: 1
I20260618 23:36:06.646627 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.646417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35347 (local address 127.0.0.1:60852)
0618 23:36:06.646511 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.646577 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260618 23:36:06.647104 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.646986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35347 (local address 127.0.0.1:60866)
0618 23:36:06.647047 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.647083 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260618 23:36:06.647264 22675 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35347: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35347: connect: Connection refused (error 111)
W20260618 23:36:06.673763 22757 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 23:36:06.674001 22757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:36:06.674019 22757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:36:06.674041 22757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:36:06.675197 22757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:36:06.675246 22757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:36:06.675261 22757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:36:06.675274 22757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:36:06.675287 22757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:36:06.676844 22757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44475,127.0.98.61:40729,127.0.98.60:35347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:36:06.677114 22757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:36:06.677376 22757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:36:06.682579 22765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:36:06.682608 22763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:36:06.682608 22762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:36:06.682829 22757 server_base.cc:1034] running on GCE node
I20260618 23:36:06.683444 22757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:36:06.683826 22757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:36:06.684983 22757 hybrid_clock.cc:648] HybridClock initialized: now 1781825766684974 us; error 34 us; skew 500 ppm
Jun 18 23:36:06 dist-test-slave-6224 krb5kdc[22594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:36:06.687840 22757 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260618 23:36:06.688364 22757 webserver.cc:460] Webserver started at http://127.0.98.60:34089/ using document root <none> and password file <none>
I20260618 23:36:06.688618 22757 fs_manager.cc:359] Metadata directory not provided
I20260618 23:36:06.688689 22757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:36:06.688841 22757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:36:06.688810 22750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } attempt: 2
I20260618 23:36:06.689399 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.689191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35347 (local address 127.0.0.1:60880)
0618 23:36:06.689303 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.689362 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260618 23:36:06.689863 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.689758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35347 (local address 127.0.0.1:60896)
0618 23:36:06.689807 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.689835 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260618 23:36:06.689961 22750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35347: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35347: connect: Connection refused (error 111)
I20260618 23:36:06.690205 22757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-2/data/instance:
uuid: "b05c2be8b43e4956881118e8d925b0fa"
format_stamp: "Formatted at 2026-06-18 23:36:06 on dist-test-slave-6224"
server_key: "75ccf15a35a0926c549ce13a4f052164"
server_key_iv: "c42e9779110f6b806e07dbb81ddc1e91"
server_key_version: "encryptionkey@0"
I20260618 23:36:06.690582 22757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-2/wal/instance:
uuid: "b05c2be8b43e4956881118e8d925b0fa"
format_stamp: "Formatted at 2026-06-18 23:36:06 on dist-test-slave-6224"
server_key: "75ccf15a35a0926c549ce13a4f052164"
server_key_iv: "c42e9779110f6b806e07dbb81ddc1e91"
server_key_version: "encryptionkey@0"
I20260618 23:36:06.692101 22757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:36:06.692986 22772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:36:06.693248 22757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:36:06.693368 22757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-2/wal
uuid: "b05c2be8b43e4956881118e8d925b0fa"
format_stamp: "Formatted at 2026-06-18 23:36:06 on dist-test-slave-6224"
server_key: "75ccf15a35a0926c549ce13a4f052164"
server_key_iv: "c42e9779110f6b806e07dbb81ddc1e91"
server_key_version: "encryptionkey@0"
I20260618 23:36:06.693447 22757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:36:06.714869 22757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:36:06.717114 22757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:36:06.717273 22757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:36:06.721866 22757 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35347
I20260618 23:36:06.721937 22824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35347 every 8 connection(s)
I20260618 23:36:06.722306 22757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-2/data/info.pb
I20260618 23:36:06.722451 22675 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } attempt: 2
I20260618 23:36:06.723460 22825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:36:06.725664 22825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:36:06.726562 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 22757
I20260618 23:36:06.726655 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825489608381-392-0/minicluster-data/master-2/wal/instance
I20260618 23:36:06.726840 392 external_mini_cluster.cc:1428] Setting key 5fe6db701f8ab8467eb6cb10652f0b4e
Jun 18 23:36:06 dist-test-slave-6224 krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 18 23:36:06 dist-test-slave-6224 krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 18 23:36:06 dist-test-slave-6224 krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260618 23:36:06.734606 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.722873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35347 (local address 127.0.0.1:60910)
0618 23:36:06.722945 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.722962 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 23:36:06.723022 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:36:06.724288 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:36:06.724290 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:36:06.724476 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:36:06.724548 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:06.724550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.725916 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.725917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:06.726654 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:06.726657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.727647 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.727648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:06.727709 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.728237 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:36:06.728243 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:36:06.729925 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:36:06.732934 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:06.732935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:06.732935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:06.733254 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:06.733582 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:06.733583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:06.733584 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:06.733663 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:06.734227 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:36:06.734229 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:36:06.734284 (+ 55us) client_negotiation.cc:769] Sending connection context
0618 23:36:06.734370 (+ 86us) client_negotiation.cc:240] Negotiation successful
0618 23:36:06.734465 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260618 23:36:06.734969 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.723143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60910 (local address 127.0.98.60:35347)
0618 23:36:06.723398 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.723415 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.723450 (+ 35us) server_negotiation.cc:407] Connection header received
0618 23:36:06.724047 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.724055 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.724117 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.724253 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.724684 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.725808 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.726770 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.727483 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.727678 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.730055 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:06.730060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:06.730061 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:06.730076 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:06.732602 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.733357 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.733361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.733362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.733493 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.733750 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.733751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.733752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.734119 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:06.734209 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:06.734469 (+ 260us) server_negotiation.cc:299] Negotiation successful
0618 23:36:06.734667 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":141,"threads_started":1}
I20260618 23:36:06.736388 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.727050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32924 (local address 127.0.98.62:44475)
0618 23:36:06.727134 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.727136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.727145 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:36:06.727165 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.727166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.727183 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.727242 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.727972 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.728530 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.729693 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.729984 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.730024 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.732933 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:06.732935 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:06.732935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:06.732944 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:06.734728 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.735458 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.735459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.735459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.735529 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.735802 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.735803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.735803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.735956 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:06.735993 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:06.736206 (+ 213us) server_negotiation.cc:299] Negotiation successful
0618 23:36:06.736291 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 23:36:06.736760 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.726425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44475 (local address 127.0.0.1:32924)
0618 23:36:06.726612 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.726737 (+ 125us) client_negotiation.cc:174] Beginning negotiation
0618 23:36:06.726809 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:36:06.727317 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:36:06.727324 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:36:06.727783 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:36:06.727876 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:06.727879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.728628 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.728629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:06.729581 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:06.729584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.730105 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.730106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:06.730181 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.730926 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:36:06.730931 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:36:06.732827 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:36:06.734795 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:06.734803 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:06.734805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:06.735365 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:06.735641 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:06.735641 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:06.735642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:06.735727 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:06.736035 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:36:06.736039 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:36:06.736087 (+ 48us) client_negotiation.cc:769] Sending connection context
0618 23:36:06.736179 (+ 92us) client_negotiation.cc:240] Negotiation successful
0618 23:36:06.736655 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":130,"threads_started":1}
I20260618 23:36:06.737488 22675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294: Bootstrap starting.
I20260618 23:36:06.738708 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.727218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60926 (local address 127.0.98.60:35347)
0618 23:36:06.727408 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.727410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.727539 (+ 129us) server_negotiation.cc:407] Connection header received
0618 23:36:06.727862 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.727863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.727875 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.727957 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.728787 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.729363 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.730271 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.731146 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.731240 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.734703 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:06.734705 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:06.734705 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:06.734712 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:06.737292 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.737746 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.737747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.737747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.737834 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.738038 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.738040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.738040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.738224 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:06.738400 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:06.738440 (+ 40us) server_negotiation.cc:299] Negotiation successful
0618 23:36:06.738531 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":167,"thread_start_us":93,"threads_started":1}
I20260618 23:36:06.738739 22675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294: Neither blocks nor log segments found. Creating new log.
I20260618 23:36:06.739099 22675 log.cc:824] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294: Log is configured to *not* fsync() on all Append() calls
I20260618 23:36:06.739961 22825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:36:06.740250 22675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294: No bootstrap required, opened a new log
I20260618 23:36:06.742223 22675 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7786b29b6184665ae5471f42dbb3294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } } peers { permanent_uuid: "239a1bc0227e400d9ba3a6c799509046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } } peers { permanent_uuid: "b05c2be8b43e4956881118e8d925b0fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } }
I20260618 23:36:06.742376 22675 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7786b29b6184665ae5471f42dbb3294, State: Initialized, Role: FOLLOWER
I20260618 23:36:06.742450 22675 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7786b29b6184665ae5471f42dbb3294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } } peers { permanent_uuid: "239a1bc0227e400d9ba3a6c799509046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } } peers { permanent_uuid: "b05c2be8b43e4956881118e8d925b0fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } }
I20260618 23:36:06.742839 22675 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:36:06.742841 22832 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7786b29b6184665ae5471f42dbb3294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } } peers { permanent_uuid: "239a1bc0227e400d9ba3a6c799509046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } } peers { permanent_uuid: "b05c2be8b43e4956881118e8d925b0fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } } }
I20260618 23:36:06.743197 22832 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294 [sys.catalog]: This master's current role is: FOLLOWER
Jun 18 23:36:06 dist-test-slave-6224 krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260618 23:36:06.744936 22843 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:36:06.744984 22843 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 18 23:36:06 dist-test-slave-6224 krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:36:06.746006 22607 master_runner.cc:186] Error getting master registration for 127.0.98.61:40729: OK, instance_id { permanent_uuid: "239a1bc0227e400d9ba3a6c799509046" instance_seqno: 1781825766539435 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 23:36:06.748963 22607 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 23:36:06.749439 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.740257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40729 (local address 127.0.0.1:35022)
0618 23:36:06.740304 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.740314 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:36:06.740350 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:36:06.740542 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:36:06.740543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:36:06.740698 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:36:06.740765 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:06.740767 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.741663 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.741664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:06.742463 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:06.742466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.743050 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.743051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:06.743124 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.743536 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:36:06.743568 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:36:06.745937 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:36:06.748150 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:06.748152 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:06.748152 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:06.748399 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:06.748606 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:06.748607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:06.748607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:06.748680 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:06.749071 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:36:06.749072 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:36:06.749127 (+ 55us) client_negotiation.cc:769] Sending connection context
0618 23:36:06.749194 (+ 67us) client_negotiation.cc:240] Negotiation successful
0618 23:36:06.749313 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260618 23:36:06.749802 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.739401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32930 (local address 127.0.98.62:44475)
0618 23:36:06.739446 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.739448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.739458 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:36:06.739483 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.739484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.739498 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.739591 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.740328 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.741031 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.742127 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.742396 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.742468 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.746052 (+ 3584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:06.746054 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:06.746054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:06.746064 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:06.748374 (+ 2310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.748805 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.748806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.748806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.748874 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.749085 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.749085 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.749086 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.749233 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:06.749381 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:06.749641 (+ 260us) server_negotiation.cc:299] Negotiation successful
0618 23:36:06.749722 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:36:06.750233 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.740341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.0.98.61:40729)
0618 23:36:06.740399 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.740401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.740410 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:36:06.740439 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.740440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.740450 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.740514 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.740866 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.741563 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.742592 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.742904 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.742953 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.746053 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:06.746054 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:06.746055 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:06.746064 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:06.748028 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.748480 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.748481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.748482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.748539 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.748748 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.748749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.748749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.748913 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:06.748985 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:06.750073 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0618 23:36:06.750160 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260618 23:36:06.750748 22825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 23:36:06 dist-test-slave-6224 krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 18 23:36:06 dist-test-slave-6224 krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260618 23:36:06.759763 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.750801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35030 (local address 127.0.98.61:40729)
0618 23:36:06.750839 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.750841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.750848 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:36:06.750944 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.750945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.750956 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.751014 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.751661 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.752226 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.753022 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.753307 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.753398 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.756595 (+ 3197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:06.756597 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:06.756598 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:06.756606 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:06.758395 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.758804 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.758807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.758807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.758891 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.759105 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.759106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.759107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.759246 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:06.759457 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:06.759495 (+ 38us) server_negotiation.cc:299] Negotiation successful
0618 23:36:06.759618 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260618 23:36:06.759850 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.751003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35347 (local address 127.0.0.1:60934)
0618 23:36:06.751089 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.751100 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:36:06.751142 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:36:06.751279 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:36:06.751280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:36:06.751462 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:36:06.751507 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:06.751509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.752564 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.752565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:06.753319 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:06.753321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.753831 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.753833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:06.753905 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.754422 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:36:06.754427 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:36:06.756435 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:36:06.758751 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:06.758752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:06.758752 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:06.758955 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:06.759180 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:06.759181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:06.759181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:06.759240 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:06.759490 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:36:06.759490 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:36:06.759535 (+ 45us) client_negotiation.cc:769] Sending connection context
0618 23:36:06.759630 (+ 95us) client_negotiation.cc:240] Negotiation successful
0618 23:36:06.759737 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260618 23:36:06.760166 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.751096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60934 (local address 127.0.98.60:35347)
0618 23:36:06.751139 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.751141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.751150 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:36:06.751197 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.751197 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.751209 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.751258 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.751668 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.752437 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.753417 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.753740 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.753815 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.756554 (+ 2739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:06.756556 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:06.756556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:06.756564 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:06.758651 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.759039 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.759040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.759041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.759101 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.759297 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.759298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.759298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.759417 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:06.759471 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:06.760019 (+ 548us) server_negotiation.cc:299] Negotiation successful
0618 23:36:06.760090 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260618 23:36:06.762248 22825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b05c2be8b43e4956881118e8d925b0fa: Bootstrap starting.
I20260618 23:36:06.763046 22825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b05c2be8b43e4956881118e8d925b0fa: Neither blocks nor log segments found. Creating new log.
I20260618 23:36:06.763378 22825 log.cc:824] T 00000000000000000000000000000000 P b05c2be8b43e4956881118e8d925b0fa: Log is configured to *not* fsync() on all Append() calls
I20260618 23:36:06.763900 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:36:06.764483 22825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b05c2be8b43e4956881118e8d925b0fa: No bootstrap required, opened a new log
I20260618 23:36:06.765317 22649 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b7786b29b6184665ae5471f42dbb3294"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32930
I20260618 23:36:06.765533 22649 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260618 23:36:06.765594 22649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:36:06.766587 22649 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7786b29b6184665ae5471f42dbb3294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } } peers { permanent_uuid: "239a1bc0227e400d9ba3a6c799509046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } } peers { permanent_uuid: "b05c2be8b43e4956881118e8d925b0fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } }
I20260618 23:36:06.766530 22825 raft_consensus.cc:348] T 00000000000000000000000000000000 P b05c2be8b43e4956881118e8d925b0fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7786b29b6184665ae5471f42dbb3294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } } peers { permanent_uuid: "239a1bc0227e400d9ba3a6c799509046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } } peers { permanent_uuid: "b05c2be8b43e4956881118e8d925b0fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } }
I20260618 23:36:06.766708 22825 raft_consensus.cc:729] T 00000000000000000000000000000000 P b05c2be8b43e4956881118e8d925b0fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b05c2be8b43e4956881118e8d925b0fa, State: Initialized, Role: FOLLOWER
I20260618 23:36:06.766810 22825 consensus_queue.cc:260] T 00000000000000000000000000000000 P b05c2be8b43e4956881118e8d925b0fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7786b29b6184665ae5471f42dbb3294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } } peers { permanent_uuid: "239a1bc0227e400d9ba3a6c799509046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } } peers { permanent_uuid: "b05c2be8b43e4956881118e8d925b0fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } }
I20260618 23:36:06.766862 22649 leader_election.cc:288] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294 [CANDIDATE]: Term 1 election: Requested vote from peers 239a1bc0227e400d9ba3a6c799509046 (127.0.98.61:40729), b05c2be8b43e4956881118e8d925b0fa (127.0.98.60:35347)
I20260618 23:36:06.767191 22825 sys_catalog.cc:564] T 00000000000000000000000000000000 P b05c2be8b43e4956881118e8d925b0fa [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:36:06.767681 22799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7786b29b6184665ae5471f42dbb3294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b05c2be8b43e4956881118e8d925b0fa"
I20260618 23:36:06.767817 22799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b05c2be8b43e4956881118e8d925b0fa [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:36:06.768987 22799 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b05c2be8b43e4956881118e8d925b0fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7786b29b6184665ae5471f42dbb3294 in term 1.
I20260618 23:36:06.769296 22626 leader_election.cc:302] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b05c2be8b43e4956881118e8d925b0fa, b7786b29b6184665ae5471f42dbb3294; no voters:
I20260618 23:36:06.769474 22832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:36:06.769651 22832 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294 [term 1 LEADER]: Becoming Leader. State: Replica: b7786b29b6184665ae5471f42dbb3294, State: Running, Role: LEADER
I20260618 23:36:06.769752 22832 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7786b29b6184665ae5471f42dbb3294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } } peers { permanent_uuid: "239a1bc0227e400d9ba3a6c799509046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } } peers { permanent_uuid: "b05c2be8b43e4956881118e8d925b0fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } }
W20260618 23:36:06.769848 22857 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b05c2be8b43e4956881118e8d925b0fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:36:06.770061 22857 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:36:06.770300 22832 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7786b29b6184665ae5471f42dbb3294. Latest consensus state: current_term: 1 leader_uuid: "b7786b29b6184665ae5471f42dbb3294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7786b29b6184665ae5471f42dbb3294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } } peers { permanent_uuid: "239a1bc0227e400d9ba3a6c799509046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } } peers { permanent_uuid: "b05c2be8b43e4956881118e8d925b0fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } } }
I20260618 23:36:06.770380 22832 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294 [sys.catalog]: This master's current role is: LEADER
I20260618 23:36:06.771535 22846 sys_catalog.cc:455] T 00000000000000000000000000000000 P b05c2be8b43e4956881118e8d925b0fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7786b29b6184665ae5471f42dbb3294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } } peers { permanent_uuid: "239a1bc0227e400d9ba3a6c799509046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } } peers { permanent_uuid: "b05c2be8b43e4956881118e8d925b0fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } } }
I20260618 23:36:06.771692 22846 sys_catalog.cc:458] T 00000000000000000000000000000000 P b05c2be8b43e4956881118e8d925b0fa [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:36:06.772485 22757 master_runner.cc:186] Error getting master registration for 127.0.98.61:40729: OK, instance_id { permanent_uuid: "239a1bc0227e400d9ba3a6c799509046" instance_seqno: 1781825766539435 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 23:36:06.774431 22757 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 23:36:06.776229 22861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:36:06.776456 22861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:36:06.781418 22750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } attempt: 3
I20260618 23:36:06.784755 22799 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b05c2be8b43e4956881118e8d925b0fa [term 1 FOLLOWER]: Refusing update from remote peer b7786b29b6184665ae5471f42dbb3294: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:36:06.785117 22832 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b05c2be8b43e4956881118e8d925b0fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Jun 18 23:36:06 dist-test-slave-6224 krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260618 23:36:06.786777 22846 sys_catalog.cc:455] T 00000000000000000000000000000000 P b05c2be8b43e4956881118e8d925b0fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7786b29b6184665ae5471f42dbb3294. Latest consensus state: current_term: 1 leader_uuid: "b7786b29b6184665ae5471f42dbb3294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7786b29b6184665ae5471f42dbb3294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } } peers { permanent_uuid: "239a1bc0227e400d9ba3a6c799509046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } } peers { permanent_uuid: "b05c2be8b43e4956881118e8d925b0fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } } }
I20260618 23:36:06.786891 22846 sys_catalog.cc:458] T 00000000000000000000000000000000 P b05c2be8b43e4956881118e8d925b0fa [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:36:06.790963 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.781746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35347 (local address 127.0.0.1:60944)
0618 23:36:06.781852 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:06.781863 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:36:06.781906 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:36:06.782100 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:36:06.782101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:36:06.782308 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:36:06.782387 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:06.782389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.783367 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.783368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:06.784201 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:06.784203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.784884 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.784885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:06.784942 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.785348 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:36:06.785354 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:36:06.787050 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:36:06.789177 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:06.789178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:06.789178 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:06.789414 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:06.790211 (+ 797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:06.790211 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:06.790212 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:06.790295 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:06.790607 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:36:06.790607 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:36:06.790662 (+ 55us) client_negotiation.cc:769] Sending connection context
0618 23:36:06.790722 (+ 60us) client_negotiation.cc:240] Negotiation successful
0618 23:36:06.790831 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260618 23:36:06.791396 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.781803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60944 (local address 127.0.98.60:35347)
0618 23:36:06.781977 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.781979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.781986 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:36:06.782005 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.782006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.782014 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.782060 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.782465 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.783277 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.784295 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.784628 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.784710 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.787156 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:06.787157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:06.787158 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:06.787165 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:06.789074 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.790062 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.790063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.790063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.790134 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.790370 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.790371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.790371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.790516 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:06.790571 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:06.791161 (+ 590us) server_negotiation.cc:299] Negotiation successful
0618 23:36:06.791276 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260618 23:36:06.793205 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.771441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60936 (local address 127.0.98.60:35347)
0618 23:36:06.771604 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.771606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.776311 (+ 4705us) server_negotiation.cc:407] Connection header received
0618 23:36:06.776410 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.776425 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.776439 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.776497 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.777323 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.778187 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.779999 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.780416 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.780500 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.784397 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:06.784399 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:06.784399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:06.784407 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:06.789964 (+ 5557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.792128 (+ 2164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.792129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.792129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.792197 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.792484 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.792484 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.792485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.792631 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:06.792716 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:06.793037 (+ 321us) server_negotiation.cc:299] Negotiation successful
0618 23:36:06.793126 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260618 23:36:06.793470 22750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 239a1bc0227e400d9ba3a6c799509046: Bootstrap starting.
I20260618 23:36:06.794415 22750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 239a1bc0227e400d9ba3a6c799509046: Neither blocks nor log segments found. Creating new log.
I20260618 23:36:06.794800 22750 log.cc:824] T 00000000000000000000000000000000 P 239a1bc0227e400d9ba3a6c799509046: Log is configured to *not* fsync() on all Append() calls
I20260618 23:36:06.796633 22859 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7786b29b6184665ae5471f42dbb3294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7786b29b6184665ae5471f42dbb3294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } } peers { permanent_uuid: "239a1bc0227e400d9ba3a6c799509046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } } peers { permanent_uuid: "b05c2be8b43e4956881118e8d925b0fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } } }
I20260618 23:36:06.796785 22859 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294 [sys.catalog]: This master's current role is: LEADER
I20260618 23:36:06.796903 22869 mvcc.cc:205] Tried to move back new op lower bound from 7298358340747059200 to 7298358340689866752. Current Snapshot: MvccSnapshot[applied={T|T < 7298358340747059200}]
I20260618 23:36:06.797897 22846 sys_catalog.cc:455] T 00000000000000000000000000000000 P b05c2be8b43e4956881118e8d925b0fa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7786b29b6184665ae5471f42dbb3294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7786b29b6184665ae5471f42dbb3294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } } peers { permanent_uuid: "239a1bc0227e400d9ba3a6c799509046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } } peers { permanent_uuid: "b05c2be8b43e4956881118e8d925b0fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } } }
I20260618 23:36:06.798003 22846 sys_catalog.cc:458] T 00000000000000000000000000000000 P b05c2be8b43e4956881118e8d925b0fa [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:36:06.802078 22861 catalog_manager.cc:1347] Generated new cluster ID: df25254780d74f0f954885014f75cb74
I20260618 23:36:06.802181 22861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:36:06.802867 22750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 239a1bc0227e400d9ba3a6c799509046: No bootstrap required, opened a new log
I20260618 23:36:06.803504 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.775924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32932 (local address 127.0.98.62:44475)
0618 23:36:06.775974 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.775976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.776518 (+ 542us) server_negotiation.cc:407] Connection header received
0618 23:36:06.777419 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.777420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.777433 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.777463 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.778191 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.778930 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.780974 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.781297 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.781359 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.796610 (+ 15251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:06.796612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:06.796612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:06.796624 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:06.799871 (+ 3247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.800632 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.800634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.800634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.800698 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.800945 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.800946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.800946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.801127 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:06.801330 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:06.803311 (+ 1981us) server_negotiation.cc:299] Negotiation successful
0618 23:36:06.803407 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260618 23:36:06.806072 22750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 239a1bc0227e400d9ba3a6c799509046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7786b29b6184665ae5471f42dbb3294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } } peers { permanent_uuid: "239a1bc0227e400d9ba3a6c799509046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } } peers { permanent_uuid: "b05c2be8b43e4956881118e8d925b0fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } }
I20260618 23:36:06.806267 22750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 239a1bc0227e400d9ba3a6c799509046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 239a1bc0227e400d9ba3a6c799509046, State: Initialized, Role: FOLLOWER
I20260618 23:36:06.806385 22750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 239a1bc0227e400d9ba3a6c799509046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7786b29b6184665ae5471f42dbb3294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } } peers { permanent_uuid: "239a1bc0227e400d9ba3a6c799509046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } } peers { permanent_uuid: "b05c2be8b43e4956881118e8d925b0fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } }
I20260618 23:36:06.806900 22750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 239a1bc0227e400d9ba3a6c799509046 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:36:06.806960 22723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 239a1bc0227e400d9ba3a6c799509046 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:36:06.808382 22723 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 239a1bc0227e400d9ba3a6c799509046 [term 1 FOLLOWER]: Refusing update from remote peer b7786b29b6184665ae5471f42dbb3294: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:36:06.808598 22724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b7786b29b6184665ae5471f42dbb3294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "239a1bc0227e400d9ba3a6c799509046"
I20260618 23:36:06.809546 22724 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 239a1bc0227e400d9ba3a6c799509046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b7786b29b6184665ae5471f42dbb3294 in term 1.
I20260618 23:36:06.810031 22832 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "239a1bc0227e400d9ba3a6c799509046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:36:06.810503 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:06.776129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35046 (local address 127.0.98.61:40729)
0618 23:36:06.776178 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.776181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.787186 (+ 11005us) server_negotiation.cc:407] Connection header received
0618 23:36:06.787324 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.787325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.787339 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.787403 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.788606 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.789357 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.790926 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:06.791222 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:06.791264 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:06.801148 (+ 9884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:06.801150 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:06.801150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:06.801159 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:06.803920 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.809508 (+ 5588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.809510 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.809511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.809584 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:06.809827 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:06.809828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:06.809828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:06.809990 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:06.810094 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:06.810319 (+ 225us) server_negotiation.cc:299] Negotiation successful
0618 23:36:06.810420 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:36:06.806838 22875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 239a1bc0227e400d9ba3a6c799509046 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7786b29b6184665ae5471f42dbb3294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } } peers { permanent_uuid: "239a1bc0227e400d9ba3a6c799509046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } } peers { permanent_uuid: "b05c2be8b43e4956881118e8d925b0fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } } }
I20260618 23:36:06.812804 22875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 239a1bc0227e400d9ba3a6c799509046 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:36:06.812939 22887 mvcc.cc:205] Tried to move back new op lower bound from 7298358340747059200 to 7298358340689866752. Current Snapshot: MvccSnapshot[applied={T|T < 7298358340747059200}]
I20260618 23:36:06.813031 22875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 239a1bc0227e400d9ba3a6c799509046 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b7786b29b6184665ae5471f42dbb3294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7786b29b6184665ae5471f42dbb3294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } } peers { permanent_uuid: "239a1bc0227e400d9ba3a6c799509046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } } peers { permanent_uuid: "b05c2be8b43e4956881118e8d925b0fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } } }
I20260618 23:36:06.813176 22875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 239a1bc0227e400d9ba3a6c799509046 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:36:06.813036 22886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 239a1bc0227e400d9ba3a6c799509046 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7786b29b6184665ae5471f42dbb3294. Latest consensus state: current_term: 1 leader_uuid: "b7786b29b6184665ae5471f42dbb3294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7786b29b6184665ae5471f42dbb3294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } } peers { permanent_uuid: "239a1bc0227e400d9ba3a6c799509046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } } peers { permanent_uuid: "b05c2be8b43e4956881118e8d925b0fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } } }
I20260618 23:36:06.813283 22886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 239a1bc0227e400d9ba3a6c799509046 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:36:06.814553 22891 catalog_manager.cc:1259] Loaded cluster ID: df25254780d74f0f954885014f75cb74
I20260618 23:36:06.814599 22891 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 239a1bc0227e400d9ba3a6c799509046: loading cluster ID for follower catalog manager: success
W20260618 23:36:06.814708 22891 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 239a1bc0227e400d9ba3a6c799509046: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260618 23:36:06.814741 22891 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:36:06.833302 22832 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b7786b29b6184665ae5471f42dbb3294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b7786b29b6184665ae5471f42dbb3294" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44475 } } peers { permanent_uuid: "239a1bc0227e400d9ba3a6c799509046" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40729 } } peers { permanent_uuid: "b05c2be8b43e4956881118e8d925b0fa" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35347 } } }
I20260618 23:36:06.833458 22832 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294 [sys.catalog]: This master's current role is: LEADER
I20260618 23:36:06.835830 22861 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:36:06.836661 22861 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:36:06.841176 22861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7786b29b6184665ae5471f42dbb3294: Generated new TSK 0
I20260618 23:36:06.841372 22861 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 23:36:06.866211 22829 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:06.865122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60954 (local address 127.0.98.60:35347)
0618 23:36:06.865173 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.865175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.865512 (+ 337us) server_negotiation.cc:407] Connection header received
0618 23:36:06.865635 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.865636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.865646 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.865676 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.866146 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60954: BlockingRecv error: recv got EOF from 127.0.0.1:60954 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260618 23:36:06.867508 22678 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:06.866569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32944 (local address 127.0.98.62:44475)
0618 23:36:06.866610 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.866612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.866968 (+ 356us) server_negotiation.cc:407] Connection header received
0618 23:36:06.867073 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.867074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.867085 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.867115 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.867444 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32944: BlockingRecv error: recv got EOF from 127.0.0.1:32944 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260618 23:36:06.868395 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:06.866844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35052 (local address 127.0.98.61:40729)
0618 23:36:06.866887 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.866889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.867864 (+ 975us) server_negotiation.cc:407] Connection header received
0618 23:36:06.867969 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.867970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.867981 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.868010 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.868324 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35052: BlockingRecv error: recv got EOF from 127.0.0.1:35052 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260618 23:36:06.871120 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:06.870222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35060 (local address 127.0.98.61:40729)
0618 23:36:06.870257 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.870259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.870491 (+ 232us) server_negotiation.cc:407] Connection header received
0618 23:36:06.870599 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.870600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.870617 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.870647 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.871075 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35060: BlockingRecv error: recv got EOF from 127.0.0.1:35060 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260618 23:36:06.874075 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:06.873349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35068 (local address 127.0.98.61:40729)
0618 23:36:06.873382 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.873384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.873585 (+ 201us) server_negotiation.cc:407] Connection header received
0618 23:36:06.873680 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.873681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.873691 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.873720 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.874029 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35068: BlockingRecv error: recv got EOF from 127.0.0.1:35068 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260618 23:36:06.878007 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:06.877221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35072 (local address 127.0.98.61:40729)
0618 23:36:06.877255 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.877258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.877464 (+ 206us) server_negotiation.cc:407] Connection header received
0618 23:36:06.877558 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.877559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.877570 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.877597 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.877962 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35072: BlockingRecv error: recv got EOF from 127.0.0.1:35072 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260618 23:36:06.884016 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:06.883152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35076 (local address 127.0.98.61:40729)
0618 23:36:06.883189 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.883191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.883412 (+ 221us) server_negotiation.cc:407] Connection header received
0618 23:36:06.883515 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.883516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.883527 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.883603 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.883966 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35076: BlockingRecv error: recv got EOF from 127.0.0.1:35076 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260618 23:36:06.894196 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:06.893299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35086 (local address 127.0.98.61:40729)
0618 23:36:06.893344 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.893346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.893594 (+ 248us) server_negotiation.cc:407] Connection header received
0618 23:36:06.893703 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.893704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.893715 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.893746 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.894138 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35086: BlockingRecv error: recv got EOF from 127.0.0.1:35086 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260618 23:36:06.912808 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:06.911802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35102 (local address 127.0.98.61:40729)
0618 23:36:06.911850 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.911852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.912121 (+ 269us) server_negotiation.cc:407] Connection header received
0618 23:36:06.912240 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.912242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.912257 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.912294 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.912729 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35102: BlockingRecv error: recv got EOF from 127.0.0.1:35102 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260618 23:36:06.946943 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:06.946213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35106 (local address 127.0.98.61:40729)
0618 23:36:06.946261 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:06.946263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:06.946383 (+ 120us) server_negotiation.cc:407] Connection header received
0618 23:36:06.946466 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:06.946467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:06.946479 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:06.946533 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:06.946858 (+ 325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35106: BlockingRecv error: recv got EOF from 127.0.0.1:35106 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260618 23:36:07.013016 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:07.012214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35108 (local address 127.0.98.61:40729)
0618 23:36:07.012344 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:07.012347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:07.012365 (+ 18us) server_negotiation.cc:407] Connection header received
0618 23:36:07.012408 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:07.012410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:07.012427 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:07.012547 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:07.012935 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35108: BlockingRecv error: recv got EOF from 127.0.0.1:35108 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260618 23:36:07.143337 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:07.142381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35120 (local address 127.0.98.61:40729)
0618 23:36:07.142532 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:07.142535 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:07.142712 (+ 177us) server_negotiation.cc:407] Connection header received
0618 23:36:07.142763 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:07.142764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:07.142778 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:07.142845 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:07.143258 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35120: BlockingRecv error: recv got EOF from 127.0.0.1:35120 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260618 23:36:07.401594 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:07.400657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35128 (local address 127.0.98.61:40729)
0618 23:36:07.400730 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:07.400732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:07.400824 (+ 92us) server_negotiation.cc:407] Connection header received
0618 23:36:07.400888 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:07.400889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:07.400904 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:07.400966 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:07.401497 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35128: BlockingRecv error: recv got EOF from 127.0.0.1:35128 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260618 23:36:07.770705 22857 catalog_manager.cc:1259] Loaded cluster ID: df25254780d74f0f954885014f75cb74
I20260618 23:36:07.770893 22857 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b05c2be8b43e4956881118e8d925b0fa: loading cluster ID for follower catalog manager: success
I20260618 23:36:07.771883 22857 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b05c2be8b43e4956881118e8d925b0fa: acquiring CA information for follower catalog manager: success
I20260618 23:36:07.772228 22857 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b05c2be8b43e4956881118e8d925b0fa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 23:36:07.816042 22891 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 239a1bc0227e400d9ba3a6c799509046: acquiring CA information for follower catalog manager: success
I20260618 23:36:07.816483 22891 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 239a1bc0227e400d9ba3a6c799509046: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 23:36:07.917924 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:07.914937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35132 (local address 127.0.98.61:40729)
0618 23:36:07.915065 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:07.915067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:07.915081 (+ 14us) server_negotiation.cc:407] Connection header received
0618 23:36:07.915191 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:07.915192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:07.915266 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:07.915357 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:36:07.915538 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:07.916125 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:07.917037 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:07.917250 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:07.917326 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:07.917471 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:36:07.917605 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:36:07.917673 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:07.917713 (+ 40us) server_negotiation.cc:299] Negotiation successful
0618 23:36:07.917746 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260618 23:36:07.922786 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:07.920247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60970 (local address 127.0.98.60:35347)
0618 23:36:07.920376 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:07.920378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:07.920384 (+ 6us) server_negotiation.cc:407] Connection header received
0618 23:36:07.920426 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:07.920426 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:07.920470 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:07.920570 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:36:07.920676 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:07.921191 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:07.921965 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:07.922167 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:07.922245 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:07.922420 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:36:07.922533 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:36:07.922626 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:07.922648 (+ 22us) server_negotiation.cc:299] Negotiation successful
0618 23:36:07.922670 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260618 23:36:07.928040 22962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:07.924902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60974 (local address 127.0.98.60:35347)
0618 23:36:07.925006 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:07.925007 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:07.925018 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:36:07.925149 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:07.925150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:07.925162 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:07.925223 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:36:07.925389 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:07.926002 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:07.927117 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:07.927435 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:07.927517 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:07.927699 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:36:07.927739 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:36:07.927793 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:07.927898 (+ 105us) server_negotiation.cc:299] Negotiation successful
0618 23:36:07.927925 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:36:07.928453 22956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:07.924922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35134 (local address 127.0.98.61:40729)
0618 23:36:07.924987 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:07.924989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:07.925121 (+ 132us) server_negotiation.cc:407] Connection header received
0618 23:36:07.925201 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:07.925202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:07.925226 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:07.925285 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:36:07.925504 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:07.926367 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:07.927363 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:07.927709 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:07.927788 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:07.928111 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:36:07.928146 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:36:07.928178 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:07.928325 (+ 147us) server_negotiation.cc:299] Negotiation successful
0618 23:36:07.928362 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 23:36:07.929075 22969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:07.924749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32958 (local address 127.0.98.62:44475)
0618 23:36:07.925202 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:07.925205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:07.925216 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:36:07.925290 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:07.925291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:07.925351 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:07.925462 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:36:07.925639 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:07.926376 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:07.927403 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:07.927718 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:07.927796 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:07.928032 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:36:07.928638 (+ 606us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:36:07.928905 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:07.928926 (+ 21us) server_negotiation.cc:299] Negotiation successful
0618 23:36:07.928964 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":66,"threads_started":1}
I20260618 23:36:07.931174 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 22607
I20260618 23:36:07.938076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 22682
I20260618 23:36:07.944046 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 22757
2026-06-18T23:36:07Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2876 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:36:07 dist-test-slave-6224 krb5kdc[22973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:36:08 dist-test-slave-6224 krb5kdc[22973](info): set up 2 sockets
Jun 18 23:36:08 dist-test-slave-6224 krb5kdc[22973](info): commencing operation
krb5kdc: starting...
W20260618 23:36:08.992165 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/test-user.keytab.
Jun 18 23:36:09 dist-test-slave-6224 krb5kdc[22973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:36:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:36:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260618 23:36:09.120633 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46027
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46027,127.0.98.61:34945,127.0.98.60:35673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:36:09.204493 22986 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 23:36:09.204761 22986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:36:09.204788 22986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:36:09.204823 22986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:36:09.205953 22986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:36:09.206030 22986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:36:09.206056 22986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:36:09.206079 22986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:36:09.206110 22986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:36:09.207625 22986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46027,127.0.98.61:34945,127.0.98.60:35673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:36:09.207906 22986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:36:09.208143 22986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:36:09.213220 22992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:36:09.213276 22994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:36:09.213243 22991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:36:09.213479 22986 server_base.cc:1034] running on GCE node
I20260618 23:36:09.213747 22986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:36:09.214059 22986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:36:09.215209 22986 hybrid_clock.cc:648] HybridClock initialized: now 1781825769215162 us; error 64 us; skew 500 ppm
Jun 18 23:36:09 dist-test-slave-6224 krb5kdc[22973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:36:09.221086 22986 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260618 23:36:09.221616 22986 webserver.cc:460] Webserver started at http://127.0.98.62:41211/ using document root <none> and password file <none>
I20260618 23:36:09.221876 22986 fs_manager.cc:359] Metadata directory not provided
I20260618 23:36:09.221956 22986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:36:09.222106 22986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:36:09.223176 22986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-0/data/instance:
uuid: "98bd2837bba44edfa40213028eb6ef54"
format_stamp: "Formatted at 2026-06-18 23:36:09 on dist-test-slave-6224"
server_key: "27234273a45cb8fb975f4d84a0fa2da8"
server_key_iv: "a93aa72a57af740eb3bff7ca35b2b01f"
server_key_version: "encryptionkey@0"
I20260618 23:36:09.223536 22986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-0/wal/instance:
uuid: "98bd2837bba44edfa40213028eb6ef54"
format_stamp: "Formatted at 2026-06-18 23:36:09 on dist-test-slave-6224"
server_key: "27234273a45cb8fb975f4d84a0fa2da8"
server_key_iv: "a93aa72a57af740eb3bff7ca35b2b01f"
server_key_version: "encryptionkey@0"
I20260618 23:36:09.225075 22986 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:36:09.225915 23001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:36:09.226174 22986 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:36:09.226286 22986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-0/wal
uuid: "98bd2837bba44edfa40213028eb6ef54"
format_stamp: "Formatted at 2026-06-18 23:36:09 on dist-test-slave-6224"
server_key: "27234273a45cb8fb975f4d84a0fa2da8"
server_key_iv: "a93aa72a57af740eb3bff7ca35b2b01f"
server_key_version: "encryptionkey@0"
I20260618 23:36:09.226375 22986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:36:09.238816 22986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:36:09.241168 22986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:36:09.241330 22986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:36:09.245838 22986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46027
I20260618 23:36:09.245901 23053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46027 every 8 connection(s)
I20260618 23:36:09.246280 22986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-0/data/info.pb
I20260618 23:36:09.246742 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 22986
I20260618 23:36:09.246878 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-0/wal/instance
I20260618 23:36:09.247167 392 external_mini_cluster.cc:1428] Setting key 0d0968598e7692d1bd7567ae8ad00782
I20260618 23:36:09.247732 23054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:36:09.250042 23054 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46027 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 18 23:36:09 dist-test-slave-6224 krb5kdc[22973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:36:09.262806 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.250914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55114 (local address 127.0.98.62:46027)
0618 23:36:09.251123 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.251128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.251203 (+ 75us) server_negotiation.cc:407] Connection header received
0618 23:36:09.251939 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.251942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.251969 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.252100 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.252719 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.254138 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.255106 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.255703 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.255845 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.259033 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:09.259039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:09.259039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:09.259047 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:09.260665 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.261223 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.261227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.261227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.261342 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.261648 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.261648 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.261649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.261889 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:09.261964 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:09.262263 (+ 299us) server_negotiation.cc:299] Negotiation successful
0618 23:36:09.262593 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":182,"server-negotiator.queue_time_us":170,"thread_start_us":97,"threads_started":1}
I20260618 23:36:09.262812 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.250860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46027 (local address 127.0.0.1:55114)
0618 23:36:09.251043 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.251090 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 23:36:09.251740 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:36:09.252126 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:36:09.252129 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:36:09.252417 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:36:09.252597 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:09.252601 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.254263 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.254263 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:09.254997 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:09.254999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.255825 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.255826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:09.255892 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.256395 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:36:09.256404 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:36:09.258921 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:36:09.260797 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:09.260803 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:09.260806 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:09.261112 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:09.261444 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:09.261445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:09.261446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:09.261567 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:09.261969 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:36:09.261976 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:36:09.262066 (+ 90us) client_negotiation.cc:769] Sending connection context
0618 23:36:09.262185 (+ 119us) client_negotiation.cc:240] Negotiation successful
0618 23:36:09.262453 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260618 23:36:09.264096 23054 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:36:09.264537 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.264400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34945 (local address 127.0.0.1:45832)
0618 23:36:09.264459 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.264512 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260618 23:36:09.264835 23002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34945: connect: Connection refused (error 111)
I20260618 23:36:09.265440 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.265319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34945 (local address 127.0.0.1:45836)
0618 23:36:09.265383 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.265419 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260618 23:36:09.265617 23054 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34945: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34945: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260618 23:36:09.275125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34945
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46027,127.0.98.61:34945,127.0.98.60:35673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:36:09.314913 23054 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } attempt: 1
I20260618 23:36:09.315587 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.315390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34945 (local address 127.0.0.1:45838)
0618 23:36:09.315469 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.315515 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260618 23:36:09.316099 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.315998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34945 (local address 127.0.0.1:45840)
0618 23:36:09.316058 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.316084 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260618 23:36:09.316215 23054 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34945: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34945: connect: Connection refused (error 111)
W20260618 23:36:09.359767 23061 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 23:36:09.359995 23061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:36:09.360013 23061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:36:09.360033 23061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:36:09.361172 23061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:36:09.361222 23061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:36:09.361235 23061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:36:09.361248 23061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:36:09.361261 23061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:36:09.362758 23061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46027,127.0.98.61:34945,127.0.98.60:35673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:36:09.363030 23061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:36:09.363266 23061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:36:09.368642 23069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:36:09.368655 23066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:36:09.368654 23067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:36:09.368916 23061 server_base.cc:1034] running on GCE node
I20260618 23:36:09.369205 23061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:36:09.369474 23061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:36:09.370628 23061 hybrid_clock.cc:648] HybridClock initialized: now 1781825769370603 us; error 52 us; skew 500 ppm
Jun 18 23:36:09 dist-test-slave-6224 krb5kdc[22973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:36:09.373136 23061 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260618 23:36:09.373610 23061 webserver.cc:460] Webserver started at http://127.0.98.61:38769/ using document root <none> and password file <none>
I20260618 23:36:09.373790 23061 fs_manager.cc:359] Metadata directory not provided
I20260618 23:36:09.373859 23061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:36:09.373960 23061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:36:09.374946 23061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-1/data/instance:
uuid: "0bbebf54df3642a685f7fee27e96a592"
format_stamp: "Formatted at 2026-06-18 23:36:09 on dist-test-slave-6224"
server_key: "0b60d5d1cb333a7da83ecc8099054bb4"
server_key_iv: "f9bfd45fb439a765bf0da104688f3617"
server_key_version: "encryptionkey@0"
I20260618 23:36:09.375240 23061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-1/wal/instance:
uuid: "0bbebf54df3642a685f7fee27e96a592"
format_stamp: "Formatted at 2026-06-18 23:36:09 on dist-test-slave-6224"
server_key: "0b60d5d1cb333a7da83ecc8099054bb4"
server_key_iv: "f9bfd45fb439a765bf0da104688f3617"
server_key_version: "encryptionkey@0"
I20260618 23:36:09.376693 23061 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:36:09.377489 23076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:36:09.377691 23061 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:36:09.377811 23061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-1/wal
uuid: "0bbebf54df3642a685f7fee27e96a592"
format_stamp: "Formatted at 2026-06-18 23:36:09 on dist-test-slave-6224"
server_key: "0b60d5d1cb333a7da83ecc8099054bb4"
server_key_iv: "f9bfd45fb439a765bf0da104688f3617"
server_key_version: "encryptionkey@0"
I20260618 23:36:09.377898 23061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:36:09.384377 23054 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } attempt: 2
I20260618 23:36:09.384943 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.384779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34945 (local address 127.0.0.1:45848)
0618 23:36:09.384859 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.384908 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260618 23:36:09.385361 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.385272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34945 (local address 127.0.0.1:45864)
0618 23:36:09.385320 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.385346 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260618 23:36:09.385486 23054 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34945: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34945: connect: Connection refused (error 111)
I20260618 23:36:09.395689 23061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:36:09.398319 23061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:36:09.398535 23061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:36:09.404070 23061 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34945
I20260618 23:36:09.404129 23128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34945 every 8 connection(s)
I20260618 23:36:09.404594 23061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-1/data/info.pb
I20260618 23:36:09.405690 23129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:36:09.407279 23129 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46027 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:36:09.411298 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 23061
I20260618 23:36:09.411448 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-1/wal/instance
I20260618 23:36:09.411722 392 external_mini_cluster.cc:1428] Setting key 214afffbe11910578214e6aab32f619e
Jun 18 23:36:09 dist-test-slave-6224 krb5kdc[22973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:36:09.417667 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.408163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55120 (local address 127.0.98.62:46027)
0618 23:36:09.408247 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.408249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.408504 (+ 255us) server_negotiation.cc:407] Connection header received
0618 23:36:09.409015 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.409016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.409031 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.409083 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.409703 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.410202 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.411214 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.411453 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.411571 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.414245 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:09.414247 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:09.414247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:09.414256 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:09.416018 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.416539 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.416541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.416542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.416609 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.416884 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.416885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.416886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.417041 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:09.417132 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:09.417423 (+ 291us) server_negotiation.cc:299] Negotiation successful
0618 23:36:09.417524 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 23:36:09.417693 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.408137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46027 (local address 127.0.0.1:55120)
0618 23:36:09.408367 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.408405 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0618 23:36:09.408891 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:36:09.409152 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:36:09.409155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:36:09.409415 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:36:09.409610 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:09.409614 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.410297 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.410298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:09.411117 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:09.411119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.411596 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.411597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:09.411723 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.412398 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:36:09.412406 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:36:09.414121 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:36:09.416111 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:09.416118 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:09.416119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:09.416447 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:09.416689 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:09.416690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:09.416690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:09.416801 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:09.417124 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:36:09.417129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:36:09.417276 (+ 147us) client_negotiation.cc:769] Sending connection context
0618 23:36:09.417366 (+ 90us) client_negotiation.cc:240] Negotiation successful
0618 23:36:09.417511 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":108,"threads_started":1}
I20260618 23:36:09.418433 23129 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 18 23:36:09 dist-test-slave-6224 krb5kdc[22973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260618 23:36:09.426662 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.418856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45874 (local address 127.0.98.61:34945)
0618 23:36:09.419062 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.419067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.419088 (+ 21us) server_negotiation.cc:407] Connection header received
0618 23:36:09.419105 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.419107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.419118 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.419207 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.419502 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.420294 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.421044 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.421363 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.421452 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.423764 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:09.423772 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:09.423772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:09.423779 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:09.425470 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.425840 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.425842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.425842 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.425895 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.426110 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.426110 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.426110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.426256 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:09.426322 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:09.426508 (+ 186us) server_negotiation.cc:299] Negotiation successful
0618 23:36:09.426582 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":145,"threads_started":1}
I20260618 23:36:09.426841 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.418758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34945 (local address 127.0.0.1:45874)
0618 23:36:09.418823 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.418837 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:36:09.418886 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:36:09.419207 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:36:09.419207 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:36:09.419362 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:36:09.419415 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:09.419416 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.420392 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.420393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:09.420952 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:09.420954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.421449 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.421449 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:09.421520 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.421987 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:36:09.421991 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:36:09.423664 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:36:09.425567 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:09.425568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:09.425569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:09.425758 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:09.425968 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:09.425969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:09.425969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:09.426040 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:09.426330 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:36:09.426331 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:36:09.426393 (+ 62us) client_negotiation.cc:769] Sending connection context
0618 23:36:09.426484 (+ 91us) client_negotiation.cc:240] Negotiation successful
0618 23:36:09.426657 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":36}
I20260618 23:36:09.427639 23129 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:36:09.428058 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.427914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35673 (local address 127.0.0.1:54098)
0618 23:36:09.427962 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.428027 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260618 23:36:09.428233 23078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35673: connect: Connection refused (error 111)
I20260618 23:36:09.428787 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.428663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35673 (local address 127.0.0.1:54108)
0618 23:36:09.428721 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.428756 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260618 23:36:09.428929 23129 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35673: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35673: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260618 23:36:09.438692 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskcnfpmz/build/release/bin/kudu
/tmp/dist-test-taskcnfpmz/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35673
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46027,127.0.98.61:34945,127.0.98.60:35673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:36:09.476672 23054 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } attempt: 3
I20260618 23:36:09.478178 23129 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } attempt: 1
I20260618 23:36:09.478626 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.478489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35673 (local address 127.0.0.1:54120)
0618 23:36:09.478549 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.478594 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260618 23:36:09.479112 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.478929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35673 (local address 127.0.0.1:54136)
0618 23:36:09.479001 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.479059 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260618 23:36:09.479274 23129 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35673: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35673: connect: Connection refused (error 111)
Jun 18 23:36:09 dist-test-slave-6224 krb5kdc[22973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260618 23:36:09.485682 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.477175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34945 (local address 127.0.0.1:45882)
0618 23:36:09.477270 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.477286 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 23:36:09.477346 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:36:09.477600 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:36:09.477601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:36:09.477780 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:36:09.477877 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:09.477879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.478884 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.478885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:09.479649 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:09.479651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.480188 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.480189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:09.480276 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.480942 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:36:09.480947 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:36:09.482504 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:36:09.484622 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:09.484623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:09.484624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:09.484809 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:09.485039 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:09.485040 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:09.485040 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:09.485095 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:09.485387 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:36:09.485388 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:36:09.485441 (+ 53us) client_negotiation.cc:769] Sending connection context
0618 23:36:09.485504 (+ 63us) client_negotiation.cc:240] Negotiation successful
0618 23:36:09.485594 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260618 23:36:09.485724 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.477315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45882 (local address 127.0.98.61:34945)
0618 23:36:09.477409 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.477413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.477426 (+ 13us) server_negotiation.cc:407] Connection header received
0618 23:36:09.477460 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.477462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.477477 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.477584 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.477971 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.478771 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.479733 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.480066 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.480153 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.482601 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:09.482602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:09.482602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:09.482610 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:09.484506 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.484895 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.484896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.484896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.484961 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.485171 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.485172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.485173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.485305 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:09.485389 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:09.485552 (+ 163us) server_negotiation.cc:299] Negotiation successful
0618 23:36:09.485634 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 23:36:09.486224 23054 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:36:09.486647 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.486532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35673 (local address 127.0.0.1:54142)
0618 23:36:09.486582 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.486628 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260618 23:36:09.487095 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.487004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35673 (local address 127.0.0.1:54156)
0618 23:36:09.487056 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.487080 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260618 23:36:09.487224 23054 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35673: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35673: connect: Connection refused (error 111)
I20260618 23:36:09.518419 23054 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } attempt: 1
I20260618 23:36:09.519071 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.518880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35673 (local address 127.0.0.1:54170)
0618 23:36:09.518970 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.519035 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260618 23:36:09.519595 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.519448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35673 (local address 127.0.0.1:54172)
0618 23:36:09.519509 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.519571 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260618 23:36:09.519737 23054 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35673: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35673: connect: Connection refused (error 111)
W20260618 23:36:09.523499 23136 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 23:36:09.523828 23136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:36:09.523865 23136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:36:09.523904 23136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:36:09.525044 23136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:36:09.525113 23136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:36:09.525139 23136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:36:09.525171 23136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:36:09.525197 23136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:36:09.526692 23136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46027,127.0.98.61:34945,127.0.98.60:35673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:36:09.526983 23136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:36:09.527231 23136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:36:09.532519 23144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:36:09.532570 23142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:36:09.532708 23136 server_base.cc:1034] running on GCE node
W20260618 23:36:09.532526 23141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:36:09.533245 23136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:36:09.533664 23136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:36:09.534902 23136 hybrid_clock.cc:648] HybridClock initialized: now 1781825769534880 us; error 42 us; skew 500 ppm
Jun 18 23:36:09 dist-test-slave-6224 krb5kdc[22973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:36:09.537919 23136 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260618 23:36:09.538548 23136 webserver.cc:460] Webserver started at http://127.0.98.60:36591/ using document root <none> and password file <none>
I20260618 23:36:09.538786 23136 fs_manager.cc:359] Metadata directory not provided
I20260618 23:36:09.538833 23136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:36:09.538933 23136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:36:09.540160 23136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-2/data/instance:
uuid: "10f6c0a913274a9ca476b30eeefa6034"
format_stamp: "Formatted at 2026-06-18 23:36:09 on dist-test-slave-6224"
server_key: "559c62b3e7612065c1bb7a3488e80b39"
server_key_iv: "c3f4171b340a9d5d66f5050431596559"
server_key_version: "encryptionkey@0"
I20260618 23:36:09.540485 23136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-2/wal/instance:
uuid: "10f6c0a913274a9ca476b30eeefa6034"
format_stamp: "Formatted at 2026-06-18 23:36:09 on dist-test-slave-6224"
server_key: "559c62b3e7612065c1bb7a3488e80b39"
server_key_iv: "c3f4171b340a9d5d66f5050431596559"
server_key_version: "encryptionkey@0"
I20260618 23:36:09.541985 23136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:36:09.542857 23151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:36:09.543105 23136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:36:09.543216 23136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-2/wal
uuid: "10f6c0a913274a9ca476b30eeefa6034"
format_stamp: "Formatted at 2026-06-18 23:36:09 on dist-test-slave-6224"
server_key: "559c62b3e7612065c1bb7a3488e80b39"
server_key_iv: "c3f4171b340a9d5d66f5050431596559"
server_key_version: "encryptionkey@0"
I20260618 23:36:09.543309 23136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:36:09.547430 23129 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } attempt: 2
I20260618 23:36:09.547996 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.547821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35673 (local address 127.0.0.1:54184)
0618 23:36:09.547895 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.547954 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260618 23:36:09.548496 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.548375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35673 (local address 127.0.0.1:54186)
0618 23:36:09.548433 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.548468 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35673: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260618 23:36:09.548661 23129 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35673: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35673: connect: Connection refused (error 111)
I20260618 23:36:09.557152 23136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:36:09.559242 23136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:36:09.559408 23136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:36:09.563700 23136 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35673
I20260618 23:36:09.563799 23203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35673 every 8 connection(s)
I20260618 23:36:09.564162 23136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-2/data/info.pb
I20260618 23:36:09.564435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskcnfpmz/build/release/bin/kudu as pid 23136
I20260618 23:36:09.564579 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskcnfpmz/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825489608381-392-0/minicluster-data/master-2/wal/instance
I20260618 23:36:09.564901 392 external_mini_cluster.cc:1428] Setting key 7fb64899cd4b0a4feb91501ea2c22113
I20260618 23:36:09.565685 23204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:36:09.568255 23204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46027 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 23:36:09 dist-test-slave-6224 krb5kdc[22973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 18 23:36:09 dist-test-slave-6224 krb5kdc[22973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:36:09.576867 23207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.565529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54188 (local address 127.0.98.60:35673)
0618 23:36:09.565731 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.565739 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.565795 (+ 56us) server_negotiation.cc:407] Connection header received
0618 23:36:09.566333 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.566340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.566393 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.566520 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.567450 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.568520 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.569582 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.570069 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.570191 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.573241 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:09.573248 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:09.573249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:09.573256 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:09.575086 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.575486 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.575490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.575490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.575618 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.575872 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.575873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.575874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.576213 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:09.576383 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:09.576474 (+ 91us) server_negotiation.cc:299] Negotiation successful
0618 23:36:09.576613 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
I20260618 23:36:09.578208 23058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.569075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55122 (local address 127.0.98.62:46027)
0618 23:36:09.569158 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.569161 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.569362 (+ 201us) server_negotiation.cc:407] Connection header received
0618 23:36:09.569400 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.569401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.569413 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.569503 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.570028 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.570561 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.571465 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.571833 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.571931 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.574720 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:09.574728 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:09.574728 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:09.574737 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:09.576556 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.576990 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.576993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.576993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.577073 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.577315 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.577316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.577317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.577490 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:09.577582 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:09.578000 (+ 418us) server_negotiation.cc:299] Negotiation successful
0618 23:36:09.578100 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 23:36:09.578233 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.568969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46027 (local address 127.0.0.1:55122)
0618 23:36:09.569255 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.569286 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0618 23:36:09.569352 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:36:09.569522 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:36:09.569524 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:36:09.569846 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:36:09.569937 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:09.569940 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.570659 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.570660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:09.571368 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:09.571371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.571986 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.571987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:09.572063 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.572605 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:36:09.572617 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:36:09.574606 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:36:09.576648 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:09.576652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:09.576654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:09.576896 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:09.577155 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:09.577156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:09.577156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:09.577243 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:09.577769 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:36:09.577779 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:36:09.577856 (+ 77us) client_negotiation.cc:769] Sending connection context
0618 23:36:09.577951 (+ 95us) client_negotiation.cc:240] Negotiation successful
0618 23:36:09.578102 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":145,"threads_started":1}
I20260618 23:36:09.578752 23204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 23:36:09 dist-test-slave-6224 krb5kdc[22973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 18 23:36:09 dist-test-slave-6224 krb5kdc[22973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260618 23:36:09.586864 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.579144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45896 (local address 127.0.98.61:34945)
0618 23:36:09.579196 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.579199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.579207 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:36:09.579234 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.579235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.579245 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.579313 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.579606 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.580193 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.581192 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.581489 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.581593 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.583561 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:09.583564 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:09.583564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:09.583588 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:09.585191 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.585647 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.585648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.585649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.585698 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.585988 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.585988 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.585988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.586099 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:09.586173 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:09.586667 (+ 494us) server_negotiation.cc:299] Negotiation successful
0618 23:36:09.586773 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260618 23:36:09.586861 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.579046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34945 (local address 127.0.0.1:45896)
0618 23:36:09.579094 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.579105 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:36:09.579141 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:36:09.579322 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:36:09.579322 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:36:09.579442 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:36:09.579499 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:09.579500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.580311 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.580312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:09.581120 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:09.581122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.581602 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.581603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:09.581702 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.582138 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:36:09.582143 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:36:09.583441 (+ 1298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:36:09.585294 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:09.585295 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:09.585296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:09.585550 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:09.585784 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:09.585785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:09.585785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:09.585910 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:09.586478 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:36:09.586480 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:36:09.586538 (+ 58us) client_negotiation.cc:769] Sending connection context
0618 23:36:09.586626 (+ 88us) client_negotiation.cc:240] Negotiation successful
0618 23:36:09.586717 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260618 23:36:09.587385 23204 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:36:09.587512 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.577766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55130 (local address 127.0.98.62:46027)
0618 23:36:09.577987 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.577989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.577996 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:36:09.578015 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.578016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.578024 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.578112 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.578756 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.579906 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.580921 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.581232 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.581313 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.584205 (+ 2892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:09.584207 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:09.584207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:09.584214 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:09.586096 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.586449 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.586450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.586451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.586509 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.586780 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.586780 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.586781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.586926 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:09.587078 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:09.587277 (+ 199us) server_negotiation.cc:299] Negotiation successful
0618 23:36:09.587364 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":104,"threads_started":1}
Jun 18 23:36:09 dist-test-slave-6224 krb5kdc[22973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 18 23:36:09 dist-test-slave-6224 krb5kdc[22973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825769, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260618 23:36:09.594874 23207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.587781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54194 (local address 127.0.98.60:35673)
0618 23:36:09.587834 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.587835 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.587844 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:36:09.587909 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.587909 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.587918 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.587993 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.588264 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.588909 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.589857 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.590180 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.590259 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.592280 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:09.592282 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:09.592282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:09.592288 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:09.593754 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.594085 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.594086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.594086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.594140 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.594334 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.594335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.594336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.594449 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:09.594523 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:09.594647 (+ 124us) server_negotiation.cc:299] Negotiation successful
0618 23:36:09.594764 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260618 23:36:09.594926 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.587733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35673 (local address 127.0.0.1:54194)
0618 23:36:09.587788 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.587800 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:36:09.587846 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:36:09.588014 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:36:09.588015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:36:09.588146 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:36:09.588178 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:09.588179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.588984 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.588986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:09.589738 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:09.589740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.590264 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.590265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:09.590310 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.590794 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:36:09.590798 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:36:09.592161 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:36:09.593834 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:09.593836 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:09.593836 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:09.594002 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:09.594204 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:09.594204 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:09.594205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:09.594265 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:09.594514 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:36:09.594515 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:36:09.594552 (+ 37us) client_negotiation.cc:769] Sending connection context
0618 23:36:09.594621 (+ 69us) client_negotiation.cc:240] Negotiation successful
0618 23:36:09.594816 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":97}
I20260618 23:36:09.594944 23054 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } attempt: 2
I20260618 23:36:09.597229 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.588429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45898 (local address 127.0.98.61:34945)
0618 23:36:09.588488 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.588490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.588500 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:36:09.588524 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.588525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.588535 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.588608 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.589077 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.589775 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.590619 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.590939 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.590999 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.593765 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:09.593766 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:09.593767 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:09.593774 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:09.595967 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.596373 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.596374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.596374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.596439 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.596683 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.596684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.596684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.596822 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:09.596994 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:09.597048 (+ 54us) server_negotiation.cc:299] Negotiation successful
0618 23:36:09.597133 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 23:36:09.597563 23204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10f6c0a913274a9ca476b30eeefa6034: Bootstrap starting.
I20260618 23:36:09.598577 23204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10f6c0a913274a9ca476b30eeefa6034: Neither blocks nor log segments found. Creating new log.
I20260618 23:36:09.598945 23204 log.cc:824] T 00000000000000000000000000000000 P 10f6c0a913274a9ca476b30eeefa6034: Log is configured to *not* fsync() on all Append() calls
I20260618 23:36:09.599768 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:36:09.600045 23204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10f6c0a913274a9ca476b30eeefa6034: No bootstrap required, opened a new log
Jun 18 23:36:09 dist-test-slave-6224 krb5kdc[22973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260618 23:36:09.600869 23028 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "98bd2837bba44edfa40213028eb6ef54"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55130
I20260618 23:36:09.602322 23204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10f6c0a913274a9ca476b30eeefa6034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bd2837bba44edfa40213028eb6ef54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46027 } } peers { permanent_uuid: "0bbebf54df3642a685f7fee27e96a592" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } } peers { permanent_uuid: "10f6c0a913274a9ca476b30eeefa6034" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } }
I20260618 23:36:09.602509 23204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10f6c0a913274a9ca476b30eeefa6034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10f6c0a913274a9ca476b30eeefa6034, State: Initialized, Role: FOLLOWER
I20260618 23:36:09.602620 23204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10f6c0a913274a9ca476b30eeefa6034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bd2837bba44edfa40213028eb6ef54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46027 } } peers { permanent_uuid: "0bbebf54df3642a685f7fee27e96a592" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } } peers { permanent_uuid: "10f6c0a913274a9ca476b30eeefa6034" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } }
I20260618 23:36:09.603086 23204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10f6c0a913274a9ca476b30eeefa6034 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:36:09.603006 23212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10f6c0a913274a9ca476b30eeefa6034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bd2837bba44edfa40213028eb6ef54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46027 } } peers { permanent_uuid: "0bbebf54df3642a685f7fee27e96a592" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } } peers { permanent_uuid: "10f6c0a913274a9ca476b30eeefa6034" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } } }
I20260618 23:36:09.603216 23212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10f6c0a913274a9ca476b30eeefa6034 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:36:09.603758 23057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.595255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35673 (local address 127.0.0.1:54202)
0618 23:36:09.595314 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.595325 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:36:09.595365 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:36:09.595905 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:36:09.595908 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:36:09.596083 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:36:09.596140 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:09.596142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.597079 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.597080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:09.597873 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:09.597876 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.598413 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.598414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:09.598472 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.598934 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:36:09.598941 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:36:09.600513 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:36:09.602406 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:09.602407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:09.602407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:09.602656 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:09.602900 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:09.602901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:09.602902 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:09.602976 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:09.603352 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:36:09.603353 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:36:09.603407 (+ 54us) client_negotiation.cc:769] Sending connection context
0618 23:36:09.603505 (+ 98us) client_negotiation.cc:240] Negotiation successful
0618 23:36:09.603651 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260618 23:36:09.603787 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.595433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54202 (local address 127.0.98.60:35673)
0618 23:36:09.595710 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.595713 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.595724 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:36:09.595748 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.595749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.595764 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.596199 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.596232 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.596982 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.598022 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.598316 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.598409 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.600614 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:09.600616 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:09.600617 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:09.600624 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:09.602292 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.602740 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.602741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.602741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.602807 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.603060 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.603061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.603061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.603271 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:09.603356 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:09.603532 (+ 176us) server_negotiation.cc:299] Negotiation successful
0618 23:36:09.603670 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":177,"threads_started":1}
I20260618 23:36:09.605587 23054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54: Bootstrap starting.
W20260618 23:36:09.605949 23223 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 10f6c0a913274a9ca476b30eeefa6034: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:36:09.606012 23223 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:36:09.606467 23054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54: Neither blocks nor log segments found. Creating new log.
I20260618 23:36:09.606767 23054 log.cc:824] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54: Log is configured to *not* fsync() on all Append() calls
I20260618 23:36:09.607759 23054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54: No bootstrap required, opened a new log
I20260618 23:36:09.606114 23136 master_runner.cc:186] Error getting master registration for 127.0.98.62:46027: OK, instance_id { permanent_uuid: "98bd2837bba44edfa40213028eb6ef54" instance_seqno: 1781825769244596 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 23:36:09.608851 23136 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 23:36:09.609171 23054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bd2837bba44edfa40213028eb6ef54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46027 } } peers { permanent_uuid: "0bbebf54df3642a685f7fee27e96a592" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } } peers { permanent_uuid: "10f6c0a913274a9ca476b30eeefa6034" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } }
I20260618 23:36:09.609313 23054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98bd2837bba44edfa40213028eb6ef54, State: Initialized, Role: FOLLOWER
I20260618 23:36:09.609400 23054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bd2837bba44edfa40213028eb6ef54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46027 } } peers { permanent_uuid: "0bbebf54df3642a685f7fee27e96a592" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } } peers { permanent_uuid: "10f6c0a913274a9ca476b30eeefa6034" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } }
I20260618 23:36:09.609750 23054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:36:09.609840 23028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260618 23:36:09.609887 23028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:36:09.611249 23028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bd2837bba44edfa40213028eb6ef54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46027 } } peers { permanent_uuid: "0bbebf54df3642a685f7fee27e96a592" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } } peers { permanent_uuid: "10f6c0a913274a9ca476b30eeefa6034" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } }
I20260618 23:36:09.611546 23028 leader_election.cc:288] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54 [CANDIDATE]: Term 1 election: Requested vote from peers 0bbebf54df3642a685f7fee27e96a592 (127.0.98.61:34945), 10f6c0a913274a9ca476b30eeefa6034 (127.0.98.60:35673)
I20260618 23:36:09.612052 23178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98bd2837bba44edfa40213028eb6ef54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "10f6c0a913274a9ca476b30eeefa6034"
I20260618 23:36:09.612205 23178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10f6c0a913274a9ca476b30eeefa6034 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:36:09.609699 23226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bd2837bba44edfa40213028eb6ef54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46027 } } peers { permanent_uuid: "0bbebf54df3642a685f7fee27e96a592" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } } peers { permanent_uuid: "10f6c0a913274a9ca476b30eeefa6034" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } } }
I20260618 23:36:09.612345 23226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:36:09.613380 23178 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 10f6c0a913274a9ca476b30eeefa6034 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98bd2837bba44edfa40213028eb6ef54 in term 1.
I20260618 23:36:09.613644 23003 leader_election.cc:302] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10f6c0a913274a9ca476b30eeefa6034, 98bd2837bba44edfa40213028eb6ef54; no voters:
I20260618 23:36:09.613811 23226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:36:09.613981 23226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54 [term 1 LEADER]: Becoming Leader. State: Replica: 98bd2837bba44edfa40213028eb6ef54, State: Running, Role: LEADER
I20260618 23:36:09.614085 23226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bd2837bba44edfa40213028eb6ef54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46027 } } peers { permanent_uuid: "0bbebf54df3642a685f7fee27e96a592" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } } peers { permanent_uuid: "10f6c0a913274a9ca476b30eeefa6034" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } }
I20260618 23:36:09.614595 23226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98bd2837bba44edfa40213028eb6ef54. Latest consensus state: current_term: 1 leader_uuid: "98bd2837bba44edfa40213028eb6ef54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bd2837bba44edfa40213028eb6ef54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46027 } } peers { permanent_uuid: "0bbebf54df3642a685f7fee27e96a592" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } } peers { permanent_uuid: "10f6c0a913274a9ca476b30eeefa6034" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } } }
I20260618 23:36:09.614679 23226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54 [sys.catalog]: This master's current role is: LEADER
I20260618 23:36:09.614514 22986 master_runner.cc:186] Error getting master registration for 127.0.98.61:34945: OK, instance_id { permanent_uuid: "0bbebf54df3642a685f7fee27e96a592" instance_seqno: 1781825769402368 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 23:36:09.617070 22986 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 23:36:09.620375 23247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:36:09.621073 23247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:36:09.624847 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.612993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55140 (local address 127.0.98.62:46027)
0618 23:36:09.613182 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.613184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.613598 (+ 414us) server_negotiation.cc:407] Connection header received
0618 23:36:09.613897 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.613898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.613909 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.613938 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.615581 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.616343 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.618117 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.618429 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.618472 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.620998 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:09.621000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:09.621000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:09.621008 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:09.622837 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.623627 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.623628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.623628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.623706 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.624039 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.624039 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.624040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.624186 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:09.624269 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:09.624613 (+ 344us) server_negotiation.cc:299] Negotiation successful
0618 23:36:09.624724 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260618 23:36:09.625059 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.613279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45900 (local address 127.0.98.61:34945)
0618 23:36:09.613349 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.613351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.614992 (+ 1641us) server_negotiation.cc:407] Connection header received
0618 23:36:09.615124 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.615127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.615144 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.615181 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.616085 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.616864 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.617999 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.618321 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.618368 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.620235 (+ 1867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:09.620236 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:09.620236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:09.620245 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:09.621987 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.623823 (+ 1836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.623824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.623825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.623894 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.624395 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.624396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.624396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.624561 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:09.624642 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:09.624849 (+ 207us) server_negotiation.cc:299] Negotiation successful
0618 23:36:09.624945 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 23:36:09.626713 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.613230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54206 (local address 127.0.98.60:35673)
0618 23:36:09.613319 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.613321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.613666 (+ 345us) server_negotiation.cc:407] Connection header received
0618 23:36:09.613695 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.613696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.613711 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.613767 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.617914 (+ 4147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.618702 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.620315 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.620975 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.621020 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.623099 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:09.623101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:09.623102 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:09.623110 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:09.624910 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.625514 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.625516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.625516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.625591 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.625815 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.625816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.625816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.625954 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:09.626058 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:09.626531 (+ 473us) server_negotiation.cc:299] Negotiation successful
0618 23:36:09.626627 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260618 23:36:09.627808 23178 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 10f6c0a913274a9ca476b30eeefa6034 [term 1 FOLLOWER]: Refusing update from remote peer 98bd2837bba44edfa40213028eb6ef54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:36:09.628230 23245 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "10f6c0a913274a9ca476b30eeefa6034" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:36:09.630352 23212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10f6c0a913274a9ca476b30eeefa6034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98bd2837bba44edfa40213028eb6ef54. Latest consensus state: current_term: 1 leader_uuid: "98bd2837bba44edfa40213028eb6ef54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bd2837bba44edfa40213028eb6ef54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46027 } } peers { permanent_uuid: "0bbebf54df3642a685f7fee27e96a592" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } } peers { permanent_uuid: "10f6c0a913274a9ca476b30eeefa6034" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } } }
I20260618 23:36:09.630461 23212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10f6c0a913274a9ca476b30eeefa6034 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:36:09.639822 23129 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } attempt: 3
I20260618 23:36:09.641417 23245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98bd2837bba44edfa40213028eb6ef54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bd2837bba44edfa40213028eb6ef54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46027 } } peers { permanent_uuid: "0bbebf54df3642a685f7fee27e96a592" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } } peers { permanent_uuid: "10f6c0a913274a9ca476b30eeefa6034" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } } }
I20260618 23:36:09.641530 23245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54 [sys.catalog]: This master's current role is: LEADER
I20260618 23:36:09.641841 23249 mvcc.cc:205] Tried to move back new op lower bound from 7298358352390459392 to 7298358352340078592. Current Snapshot: MvccSnapshot[applied={T|T < 7298358352390459392}]
I20260618 23:36:09.642345 23212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10f6c0a913274a9ca476b30eeefa6034 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98bd2837bba44edfa40213028eb6ef54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bd2837bba44edfa40213028eb6ef54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46027 } } peers { permanent_uuid: "0bbebf54df3642a685f7fee27e96a592" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } } peers { permanent_uuid: "10f6c0a913274a9ca476b30eeefa6034" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } } }
I20260618 23:36:09.642464 23212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10f6c0a913274a9ca476b30eeefa6034 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:36:09.642495 23247 catalog_manager.cc:1347] Generated new cluster ID: d556702bdf09409bb2200ce1ecbe55fc
I20260618 23:36:09.642545 23247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:36:09.643620 23250 mvcc.cc:205] Tried to move back new op lower bound from 7298358352390459392 to 7298358352340078592. Current Snapshot: MvccSnapshot[applied={T|T < 7298358352390459392}]
Jun 18 23:36:09 dist-test-slave-6224 krb5kdc[22973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260618 23:36:09.649762 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.640239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35673 (local address 127.0.0.1:54214)
0618 23:36:09.640299 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0618 23:36:09.640311 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:36:09.640358 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:36:09.640580 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:36:09.640581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:36:09.640746 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:36:09.640789 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:09.640791 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.641694 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.641695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:09.642442 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:36:09.642445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.643850 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.643851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:36:09.643908 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.644400 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:36:09.644405 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:36:09.645904 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:36:09.647885 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:09.647886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:09.647886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:09.648095 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:09.649043 (+ 948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:36:09.649044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:36:09.649044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:36:09.649110 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:36:09.649469 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:36:09.649470 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:36:09.649522 (+ 52us) client_negotiation.cc:769] Sending connection context
0618 23:36:09.649575 (+ 53us) client_negotiation.cc:240] Negotiation successful
0618 23:36:09.649663 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260618 23:36:09.650106 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:09.640265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54214 (local address 127.0.98.60:35673)
0618 23:36:09.640444 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.640446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.640454 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:36:09.640476 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.640477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.640488 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.640533 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.640901 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.641579 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.642563 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:09.642859 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:09.642916 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:09.646026 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:36:09.646027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:36:09.646027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:36:09.646034 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:36:09.647779 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.648899 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.648899 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.648900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.648968 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:36:09.649173 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:36:09.649174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:36:09.649174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:36:09.649385 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:36:09.649434 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:09.649941 (+ 507us) server_negotiation.cc:299] Negotiation successful
0618 23:36:09.650025 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260618 23:36:09.651455 23129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bbebf54df3642a685f7fee27e96a592: Bootstrap starting.
I20260618 23:36:09.652215 23129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bbebf54df3642a685f7fee27e96a592: Neither blocks nor log segments found. Creating new log.
I20260618 23:36:09.652518 23129 log.cc:824] T 00000000000000000000000000000000 P 0bbebf54df3642a685f7fee27e96a592: Log is configured to *not* fsync() on all Append() calls
I20260618 23:36:09.656260 23129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bbebf54df3642a685f7fee27e96a592: No bootstrap required, opened a new log
I20260618 23:36:09.658129 23129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bbebf54df3642a685f7fee27e96a592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bd2837bba44edfa40213028eb6ef54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46027 } } peers { permanent_uuid: "0bbebf54df3642a685f7fee27e96a592" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } } peers { permanent_uuid: "10f6c0a913274a9ca476b30eeefa6034" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } }
I20260618 23:36:09.658262 23129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bbebf54df3642a685f7fee27e96a592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bbebf54df3642a685f7fee27e96a592, State: Initialized, Role: FOLLOWER
I20260618 23:36:09.658344 23129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bbebf54df3642a685f7fee27e96a592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bd2837bba44edfa40213028eb6ef54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46027 } } peers { permanent_uuid: "0bbebf54df3642a685f7fee27e96a592" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } } peers { permanent_uuid: "10f6c0a913274a9ca476b30eeefa6034" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } }
I20260618 23:36:09.658660 23129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bbebf54df3642a685f7fee27e96a592 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:36:09.659770 23103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98bd2837bba44edfa40213028eb6ef54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0bbebf54df3642a685f7fee27e96a592"
I20260618 23:36:09.659879 23103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bbebf54df3642a685f7fee27e96a592 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:36:09.660655 23103 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0bbebf54df3642a685f7fee27e96a592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98bd2837bba44edfa40213028eb6ef54 in term 1.
I20260618 23:36:09.661726 23102 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0bbebf54df3642a685f7fee27e96a592 [term 1 FOLLOWER]: Refusing update from remote peer 98bd2837bba44edfa40213028eb6ef54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:36:09.661994 23226 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bbebf54df3642a685f7fee27e96a592" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:36:09.667791 23255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bbebf54df3642a685f7fee27e96a592 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98bd2837bba44edfa40213028eb6ef54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bd2837bba44edfa40213028eb6ef54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46027 } } peers { permanent_uuid: "0bbebf54df3642a685f7fee27e96a592" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } } peers { permanent_uuid: "10f6c0a913274a9ca476b30eeefa6034" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } } }
I20260618 23:36:09.667893 23255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bbebf54df3642a685f7fee27e96a592 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:36:09.667948 23255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bbebf54df3642a685f7fee27e96a592 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98bd2837bba44edfa40213028eb6ef54. Latest consensus state: current_term: 1 leader_uuid: "98bd2837bba44edfa40213028eb6ef54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bd2837bba44edfa40213028eb6ef54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46027 } } peers { permanent_uuid: "0bbebf54df3642a685f7fee27e96a592" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } } peers { permanent_uuid: "10f6c0a913274a9ca476b30eeefa6034" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } } }
I20260618 23:36:09.668004 23255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bbebf54df3642a685f7fee27e96a592 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:36:09.668038 23255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bbebf54df3642a685f7fee27e96a592 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "98bd2837bba44edfa40213028eb6ef54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bd2837bba44edfa40213028eb6ef54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46027 } } peers { permanent_uuid: "0bbebf54df3642a685f7fee27e96a592" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } } peers { permanent_uuid: "10f6c0a913274a9ca476b30eeefa6034" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } } }
I20260618 23:36:09.668087 23255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bbebf54df3642a685f7fee27e96a592 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:36:09.668675 23266 mvcc.cc:205] Tried to move back new op lower bound from 7298358352390459392 to 7298358352340078592. Current Snapshot: MvccSnapshot[applied={T|T < 7298358352390459392}]
I20260618 23:36:09.673216 23245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "98bd2837bba44edfa40213028eb6ef54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "98bd2837bba44edfa40213028eb6ef54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46027 } } peers { permanent_uuid: "0bbebf54df3642a685f7fee27e96a592" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34945 } } peers { permanent_uuid: "10f6c0a913274a9ca476b30eeefa6034" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35673 } } }
I20260618 23:36:09.673309 23245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54 [sys.catalog]: This master's current role is: LEADER
I20260618 23:36:09.673481 23270 catalog_manager.cc:1259] Loaded cluster ID: d556702bdf09409bb2200ce1ecbe55fc
I20260618 23:36:09.673519 23270 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0bbebf54df3642a685f7fee27e96a592: loading cluster ID for follower catalog manager: success
W20260618 23:36:09.673601 23270 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 0bbebf54df3642a685f7fee27e96a592: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260618 23:36:09.673629 23270 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:36:09.674839 23247 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:36:09.675675 23247 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:36:09.689509 23247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98bd2837bba44edfa40213028eb6ef54: Generated new TSK 0
I20260618 23:36:09.689790 23247 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 23:36:09.718189 23209 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:09.716453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55150 (local address 127.0.98.62:46027)
0618 23:36:09.716497 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.716499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.717139 (+ 640us) server_negotiation.cc:407] Connection header received
0618 23:36:09.717416 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.717417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.717430 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.717463 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.718081 (+ 618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55150: BlockingRecv error: recv got EOF from 127.0.0.1:55150 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260618 23:36:09.718737 23210 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:09.716198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54216 (local address 127.0.98.60:35673)
0618 23:36:09.716261 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.716263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.717047 (+ 784us) server_negotiation.cc:407] Connection header received
0618 23:36:09.717535 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.717536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.717548 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.717575 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.718666 (+ 1091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54216: BlockingRecv error: recv got EOF from 127.0.0.1:54216 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260618 23:36:09.719435 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:09.715937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45902 (local address 127.0.98.61:34945)
0618 23:36:09.715997 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.716000 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.717316 (+ 1316us) server_negotiation.cc:407] Connection header received
0618 23:36:09.718949 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.718950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.718962 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.718992 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.719353 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45902: BlockingRecv error: recv got EOF from 127.0.0.1:45902 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260618 23:36:09.724704 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:09.723899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45918 (local address 127.0.98.61:34945)
0618 23:36:09.723967 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.723970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.724143 (+ 173us) server_negotiation.cc:407] Connection header received
0618 23:36:09.724276 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.724277 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.724299 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.724331 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.724650 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45918: BlockingRecv error: recv got EOF from 127.0.0.1:45918 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260618 23:36:09.732445 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:09.727070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45922 (local address 127.0.98.61:34945)
0618 23:36:09.727108 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.727110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.731859 (+ 4749us) server_negotiation.cc:407] Connection header received
0618 23:36:09.731968 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.731969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.731981 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.732010 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.732380 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45922: BlockingRecv error: recv got EOF from 127.0.0.1:45922 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260618 23:36:09.736672 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:09.735787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45938 (local address 127.0.98.61:34945)
0618 23:36:09.735864 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.735866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.736123 (+ 257us) server_negotiation.cc:407] Connection header received
0618 23:36:09.736237 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.736238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.736250 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.736280 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.736618 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45938: BlockingRecv error: recv got EOF from 127.0.0.1:45938 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260618 23:36:09.742856 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:09.741981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45944 (local address 127.0.98.61:34945)
0618 23:36:09.742023 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.742025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.742284 (+ 259us) server_negotiation.cc:407] Connection header received
0618 23:36:09.742391 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.742392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.742404 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.742434 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.742799 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45944: BlockingRecv error: recv got EOF from 127.0.0.1:45944 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260618 23:36:09.753376 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:09.752363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45950 (local address 127.0.98.61:34945)
0618 23:36:09.752416 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.752418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.752695 (+ 277us) server_negotiation.cc:407] Connection header received
0618 23:36:09.752813 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.752814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.752826 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.752859 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.753303 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45950: BlockingRecv error: recv got EOF from 127.0.0.1:45950 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260618 23:36:09.772292 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:09.771136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45952 (local address 127.0.98.61:34945)
0618 23:36:09.771190 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.771193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.771525 (+ 332us) server_negotiation.cc:407] Connection header received
0618 23:36:09.771673 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.771675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.771692 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.771734 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.772203 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45952: BlockingRecv error: recv got EOF from 127.0.0.1:45952 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260618 23:36:09.807173 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:09.806011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45958 (local address 127.0.98.61:34945)
0618 23:36:09.806069 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.806072 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.806363 (+ 291us) server_negotiation.cc:407] Connection header received
0618 23:36:09.806481 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.806482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.806496 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.806528 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.807069 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45958: BlockingRecv error: recv got EOF from 127.0.0.1:45958 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260618 23:36:09.873478 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:09.872662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45966 (local address 127.0.98.61:34945)
0618 23:36:09.872743 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:09.872746 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:09.872881 (+ 135us) server_negotiation.cc:407] Connection header received
0618 23:36:09.872924 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:09.872925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:09.872936 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:09.873034 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:09.873402 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45966: BlockingRecv error: recv got EOF from 127.0.0.1:45966 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260618 23:36:10.004191 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:10.003199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.0.98.61:34945)
0618 23:36:10.003271 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:10.003273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:10.003281 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:36:10.003309 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:10.003310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:10.003320 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:10.003389 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:10.004107 (+ 718us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45982: BlockingRecv error: recv got EOF from 127.0.0.1:45982 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260618 23:36:10.262477 23133 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:36:10.261621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45990 (local address 127.0.98.61:34945)
0618 23:36:10.261709 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:10.261711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:10.261807 (+ 96us) server_negotiation.cc:407] Connection header received
0618 23:36:10.261850 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:10.261851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:10.261862 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:10.261928 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:36:10.262427 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45990: BlockingRecv error: recv got EOF from 127.0.0.1:45990 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 23:36:10.606668 23223 catalog_manager.cc:1259] Loaded cluster ID: d556702bdf09409bb2200ce1ecbe55fc
I20260618 23:36:10.606818 23223 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 10f6c0a913274a9ca476b30eeefa6034: loading cluster ID for follower catalog manager: success
I20260618 23:36:10.607769 23223 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 10f6c0a913274a9ca476b30eeefa6034: acquiring CA information for follower catalog manager: success
I20260618 23:36:10.608121 23223 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 10f6c0a913274a9ca476b30eeefa6034: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 23:36:10.674858 23270 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0bbebf54df3642a685f7fee27e96a592: acquiring CA information for follower catalog manager: success
I20260618 23:36:10.675191 23270 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0bbebf54df3642a685f7fee27e96a592: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 23:36:10.778973 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:10.775962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45998 (local address 127.0.98.61:34945)
0618 23:36:10.776112 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:10.776114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:10.776206 (+ 92us) server_negotiation.cc:407] Connection header received
0618 23:36:10.776248 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:10.776249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:10.776318 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:10.776440 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:36:10.776658 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:10.777196 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:10.778140 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:10.778329 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:10.778381 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:10.778558 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:36:10.778686 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:36:10.778753 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:10.778807 (+ 54us) server_negotiation.cc:299] Negotiation successful
0618 23:36:10.778828 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260618 23:36:10.783807 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:10.781386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54230 (local address 127.0.98.60:35673)
0618 23:36:10.781529 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:10.781531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:10.781538 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:36:10.781580 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:10.781581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:10.781622 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:10.781713 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:36:10.781813 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:10.782320 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:10.783051 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:10.783242 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:10.783296 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:10.783412 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:36:10.783527 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:36:10.783666 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:10.783713 (+ 47us) server_negotiation.cc:299] Negotiation successful
0618 23:36:10.783733 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
I20260618 23:36:10.789228 23350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:10.785904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.0.98.62:46027)
0618 23:36:10.786086 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:10.786088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:10.786230 (+ 142us) server_negotiation.cc:407] Connection header received
0618 23:36:10.786297 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:10.786298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:10.786356 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:10.786469 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:36:10.786687 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:10.787196 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:10.788114 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:10.788352 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:10.788410 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:10.788629 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:36:10.788745 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:36:10.788802 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:10.788930 (+ 128us) server_negotiation.cc:299] Negotiation successful
0618 23:36:10.789088 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":61,"threads_started":1}
I20260618 23:36:10.789273 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:10.785852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46004 (local address 127.0.98.61:34945)
0618 23:36:10.786019 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:10.786022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:10.786279 (+ 257us) server_negotiation.cc:407] Connection header received
0618 23:36:10.786370 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:10.786371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:10.786383 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:10.786436 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:36:10.786683 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:10.787397 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:10.788504 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:10.788733 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:10.788815 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:10.788978 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:36:10.789018 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:36:10.789054 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:10.789165 (+ 111us) server_negotiation.cc:299] Negotiation successful
0618 23:36:10.789189 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260618 23:36:10.789415 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:36:10.786065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54244 (local address 127.0.98.60:35673)
0618 23:36:10.786111 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0618 23:36:10.786113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:36:10.786122 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:36:10.786150 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:36:10.786151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:36:10.786167 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:36:10.786241 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:36:10.786442 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:10.787241 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:10.788455 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:36:10.788750 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:36:10.788817 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:36:10.789005 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:36:10.789045 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:36:10.789110 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0618 23:36:10.789169 (+ 59us) server_negotiation.cc:299] Negotiation successful
0618 23:36:10.789192 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260618 23:36:10.791131 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 22986
I20260618 23:36:10.798043 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 23061
I20260618 23:36:10.804718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskcnfpmz/build/release/bin/kudu with pid 23136
2026-06-18T23:36:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2861 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5738 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (281200 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST